sregistry pull does not solve the url correctly about sregistry

[vsoch]

Hey @victorsndvg !I am not able to reproduce this error - the first call to get the "result" object is actually grabbing an image manifest (I'll rename so it's more clear):

url
'http://127.0.0.1/api/container/vanessa/squashy:latest'

then this is what the manifest looks like:

{'add_date': '2017-10-10T07:11:06.839906-05:00',
 'collection': 'vanessa',
 'frozen': False,
 'id': 1,
 'image': 'http://127.0.0.1/containers/1/download/da90bd8a-7b3a-454c-bed8-d91ede223f16',
 'metadata': {'deffile': 'Bootstrap: docker\nFrom: ubuntu\n\n%help\nHeeeelp!\n\n%environment\n    VADER=badguy\n    LUKE=goodguy\n    SOLO=someguy\n    export VADER LUKE SOLO\n\n%labels\n   Maintainer Vanessasaur\n\n##############################\n# foo\n##############################\n\n%apprun foo\n    exec echo "RUNNING FOO"\n\n%applabels foo\n   BESTAPP=FOO\n   export BESTAPP\n\n%appinstall foo\n   touch foo.exec\n\n%appenv foo\n    SOFTWARE=foo\n    export SOFTWARE\n\n%apphelp foo\n    This is the help for foo.\n\n%appfiles foo\n   avocados.txt\n\n\n##############################\n# bar\n##############################\n\n%apphelp bar\n    This is the help for bar.\n\n%applabels bar\n   BESTAPP=BAR\n   export BESTAPP\n\n%appinstall bar\n    touch bar.exec\n\n%appenv bar\n    SOFTWARE=bar\n    export SOFTWARE\n\n%runscript\nexec echo "Try running with --app dog/cat"\n\n%apprun dog\nexec echo Hello "$@", this is Dog\n\n%apprun cat\nexec echo Meow "$@", this is Cat\n',
  'environment': '# Custom environment shell code should follow\n\n    VADER=badguy\n    LUKE=goodguy\n    SOLO=someguy\n    export VADER LUKE SOLO\n\n',
  'runscript': '#!/bin/sh \n\nexec echo "Try running with --app dog/cat"\n\n',
  'size_mb': 44,
  'test': None},
 'metrics': {},
 'name': 'squashy',
 'tag': 'latest',
 'version': None}

Note that the key 'image' is where the url lives to download it

'http://127.0.0.1/containers/1/download/da90bd8a-7b3a-454c-bed8-d91ede223f16'

If you specify url , you would just be downloading the manifest and renaming it to be an image file! I just tested this (not able to reproduce) and added more verbosity to the debug mode to test. Would you mind running:

sregistry --debug pull vanessa/ubuntu

and if it's a squash fs (singularity 2.4 image) run:

sregistry --debug pull vanessa/squashy --no-decompress

if we can get some info about what the call is returning for you (hence the key image is not present) we can fix it up!

[victorsndvg]

mmmm Ok,

now I'm not in front of the PC with sregistry installation. In my case I get an empty manifest (result value is {}).

tomorrow (for me), I will extend the info about this issue.

[vsoch]

okay great! I'll be around when you are ready to debug - the first easy thing to do would be to check what you see at the URL where the manifest should be. It's likely some permissions / path error.

[victorsndvg]

Good morning @vsoch ,

I think I misunderstood the usage of sregistry pull.

The command was failing is:

$ sregistry pull shub:/X.X.X.X/myname/myimage:mytag

Where X.X.X.X is the sregistry web service IP address

This following commands run perfectly in the host serving the sregistry web service:

$ sregistry push image.img --name myname/myimage --tag mytag
$ sregistry pull myname/myimage:mytag

But know I'm not able to get this image from a remote computer. I figure out the right command to do that is the following:

$ sregistry pull shub:/X.X.X.X/myname/myimage:mytag

or

$ singularity pull shub:/X.X.X.X/myname/myimage:mytag

I would like to build a "registry of registries" in which each one of them can be federated and can contain different images. In the case that a requested image from one of this federated hosts is not available within the proper host, but available in the "registry of registries", it can be retrieved (by means of sregistry|singularity pull command.

Do you think this is possible? How can I do that using sregistry or singularity?

Thanks in advance,
víctor.

[vsoch]

hey @victorsndvg - you found the syntax! Actually, the first uri (shub://) works like a url, so you would want two slahes:

sregistry pull shub://X.X.X.X/myname/myimage:mytag

that may have caused you some headache if the uri wasn't properly removed using one slash.

That's a really cool idea - right now, we have the following convention:

• sregistry pull shub://X.X.X.X/myname/myimage:mytag specification of a registry URI means using it. In this example, we use your registry at host X.X.X.X
• sregistry pull shub://myname/myimage:mytag Not specifying a host (with sregistry) uses the default set in your config file. If you are using the default and on a remote, it's going to try localhost (and not work) so you would want to have the config file set to the actual host (and not localhost)
• singularity pull shub//:X.X.X.X/myname/myimage:mytag This will work like the first, but using singularity software proper. If you are on your local machine and use localhost, if the registry is at localhost it should work ok.
• singularity pull shub://myname/myimage:mytag This is not going to hit your registry. It will use the default set in singularity, which is for singularity hub proper.

Then what you are asking is basically "let me provide a URI and search across registries." I do think this is possible, but I'm not sure the best way to implement it. It might be challenging, off the bat, to get into singularity proper because arguably if the user is allowed to ask for a URI and get two different containers with the same call, this breaks reproducibility. But I can chat about what I've done so far and we can figure out some options.

Registry of Registries

I thought ahead and already have created this - and it's exactly that! When you make a registry you register it here, and the API call to expose all public container collections is rendered: https://singularityhub.github.io/containers/registry/randomroad-registry/. This is still a pretty insular strategy in that there is no formal database - given many registries it would mean a call and parsing of each of their endpoints to answer the question "does this registry have this image" and then you'd have to decide between them.

How is a list maintained?

The simplest (and not so great) implementation I can think of is that this resource linked above could minimally serve an api endpoint to expose the different registry hosts. Then a call from the user to "search?" "find?" "locate?" a particular uri could ping the different endpoints, and bring back a list of results. Eg:

sregistry search hello-world

**Searching for containers across 3 registries...**
127.0.0.1/vsoch/hello-world:latest
127.0.0.1/vsoch/hello-world:apps
172.2.2.01/vsoch/hello-world:v20

But wourld we ask the user to still specify a custom?

sregistry pull shub://172.2.2.01/vsoch/hello-world:v20  # or
singularity pull shub://172.2.2.01/vsoch/hello-world:v20

And then what about registries that aren't listed? Would that be a feature for just public ones (and thus encourage others to share / register their registries?)

What I don't think make sense is to make decisions on behalf of the user, eg, automatically pull one from the list above. The reason we cannot make an assumption that images are equivalent based on URI is that they probably are not. There are a few different use cases:

• if we look for truly identical images, we probably want to look at versions, which are file hashes for the build. It's unlikely to find much overlap, because a registry would have to have grabbed and then pushed the exact same image.
• if we look for "images like mine" some kind of search would be appropriate, but the user would still need to choose. As I mentioned above, it's unlikely to find exact, more likely to find similar.

What are your thoughts? I'd like to implement something for this use case - being able to share and then search across (or find identical). I'll definitely put more thought into it - right now my gut is that we need some kind of addition to the "Registry of Registries" I linked above, but I'm not sure the best (meaning efficient) way to parse that, at least without a database proper.

[victorsndvg]

Hi @vsoch ,

Ops, the single slash was a typo :|

Ok, now the convention is clear. I understand I should be able to connect to a remote sregistry using sregistry or singularity commands with the right URI.

About the registry of registries, I'm thinking in a single/central point where to submit new images. With this approach image identifiers will be unique (from this central point), but can be repeated through several registries. The same identifier univocally identifies a container.

In our project all registries will be public, but images can be public or private. The particular architecture of our project will manage only one registry per Cluster, but will expose a single "Market" to end-users.They will choose an application and a particular Cluster where to launch the container. Users should not be aware of this architecture.

Then, If the requested image is in the general registry, but not into a particular cluster registry, the platform will automagically transfer the image (push/pull) between registries to have it "locally" available.

I understand that this objective could not be your objective, and probably we will need to implement a intermediate layer to manage this kind of usage. What I want is to explore your tools, to understand how to use them and to discover which already present features are useful for us an which new features could be interesting for you.

In general, I think I'm describing a scalable-and-distributed-single-registry, but with some tunning for our particular needs. Now I'm in a preliminary status exploring your tools and probably I will make a lot of mistakes and ask for stupid things ... ;P

What do you think about that?
Which features could be interesting for you?

Hope to have shown a clearest view of my needs.

Thanks for your explanations,
Víctor.

[vsoch]

hey @victorsndvg ! I think this "central" registry that you want exists - the predecessor to singularity registry is singularity hub: https://singularity-hub.org (and version 2.0 to come out soon!) However it doesn't give anyone push access directly - it's done via Github / webhooks / builder (we build the images for the users from recipe files).

I think I understand your use case - and it's cool! The idea is that a cluster of registries all know about one another, and then go along some random (or ordered) list to query if an image is found. What I don't quite understand is why, given that a user can pull from a registry, it would be needed to push the image to a second registry. You can save the redundant pull --> push --> pull --> use by just doing pull --> use.

When you say cluster - you don't mean that the registry will be installed there? it's dockerized, so that might not be a great thing to have on a shared resource.

I think I do see the vision, and also I think this is very important. It's this idea of communication between registries, easy sharing of images. Even a simple coordinating of a request --> pull --> push from some external worker would make sense. Let me think about it? I don't have the bandwidth to develop this atm, but after the release of Singularity Hub 2.0 and finishing up Singularity 2.4, let's work on this yes?

[victorsndvg]

Hey @vsoch,

the user will not be the one in charge of pulling and image. Is the infrastructure itself who will do this task. The user can decide any involved cluster to launch a containerized app. If it's the first time an image is used in a cluster it will be transfered to it. Pros of this strategy is that it helps to avoid data transfers, at most 1 transfer per image and cluster. The cons is that some data will be duplicated.

It has more sense now?

When is expected to be released these new versions?

I'm going to continue playing with your tools and submit my thoughts.

Thanks again for your help :)

[vsoch]

the user will not be the one in charge of pulling and image. Is the infrastructure itself who will do this task. The user can decide any involved cluster to launch a containerized app. If it's the first time an image is used in a cluster it will be transfered to it. Pros of this strategy is that it helps to avoid data transfers, at most 1 transfer per image and cluster. The cons is that some data will be duplicated.

so this is basically like kubernetes, but for singularity, and the containers come from shared registries?

When is expected to be released these new versions?

When I get it done. I'm not sleeping and working around the clock, I can't say much more than that.

I'm going to continue playing with your tools and submit my thoughts.
Thanks again for your help :)

Awesome! And thanks for your feedback! Again, we can address this idea more when I have some of the current finished up.

[victorsndvg]

I don't know too much about docker and kubernetes ... but I will read more to understand and learn what they do!

After this large talk ;) I think we can close this issue... And of course, keep in touch to talk deeper about this idea.

Thanks!