singularityhub / sregistry Goto Github PK

Hi,

I would like to know if it's possible to share a collection or image by a groups of users. This means that some user-defined groups of users could have permission to access to private images.

How do you deal with this?
Is possible to do that right now?
If not, is there a workaround to allow custom non-owner users of a private image to work with it?

Thanks in advance!

Is there a limit on how big the 'labels' cell can get in the table on the 'collections' page? If you put a lot of images into a collection it seems to grow rather big. E.g. if I put all of the biocontainers (>1000) into a single collection would it be huge? Looks like there's a label for every distinct 'From:' in conatiners within the collection.

Not sure how would be best to deal with this.

Hi @ALL,

in credentials I identify 2 kind of roles, superuser and admin, but I don't see clear the difference between them.

What can and cannot a superuser/admin do?
There are more roles?

I would like to have a paragraph in the documentation adding some information from this point of view.

What do you think?

Thanks in advance!

This might actually be good use for a kind of tree map:

https://bl.ocks.org/mbostock/e6962a152275373f8504

Hi!
an idea we discussed on our retreat recently was the possibility to have DOIs for containers within a certain registry. This would make it possible to state e.g. in a publication:

• We utilized software X in container Y (DOI: XXXX/YYYY/ZZZZ)

And thus be even consistently able to tell someone how you did a certain kind of analysis (when you also include parameters etc.). Would this be feasible at all or maybe feasible in Singularity hub (not sure whether this is the right place to ask for it?)

Hi,

deploying the entire infrastructure or piece by piece and taking a look to the logs I can see some errors related with users_user table (I think).

uwsgi_1   | django.db.utils.ProgrammingError: relation "users_user" does not exist

db_1      | 2017-10-19 11:13:14.396 UTC [63] ERROR:  relation "users_user" does not exist at character 397
db_1      | 2017-10-19 11:13:14.396 UTC [63] STATEMENT:  SELECT "users_user"."id", "users_user"."password", "users_user"."last_login", "users_user"."is_superuser", "users_user"."username", "users_user"."first_name", "users_user"."last_name", "users_user"."email", "users_user"."is_staff", "users_user"."is_active", "users_user"."date_joined", "users_user"."active", "users_user"."admin", "users_user"."agree_terms", "users_user"."agree_terms_date" FROM "users_user" WHERE "users_user"."username" = 'AnonymousUser'
db_1      | 2017-10-19 11:15:31.119 UTC [68] ERROR:  relation "users_user" does not exist at character 397

What do you think?

imagine - a cluster of known singularity registry that have (some) shared metadata, and are able to share containers --> https://ipfs.io/#how

Created a container, centos.img by using singularity bootstrap on the definition in the Singularity examples/centos directory. This is a container bootstrapped with yum, not imported from docker. Cannot push this container to sregistry, it fails with an exception.

Pushing a container build from examples/docker is okay, as are other containers built from docker containers. Maybe to do with the bootstrap centos container not having a From: value

20:56 $ sregistry push --name examples/centos centos.img
WARNING Cannot load metadata to parse From: line.
Compressing image Traceback (most recent call last):
  File "/home/dave/.local/bin/sregistry", line 9, in <module>
    load_entry_point('singularity==1.2.0', 'console_scripts', 'sregistry')()
  File "/home/dave/.local/lib/python2.7/site-packages/singularity-1.2.0-py2.7.egg/singularity/registry/main/__init__.py", line 204, in main
    subparser=subparsers[args.command])
  File "/home/dave/.local/lib/python2.7/site-packages/singularity-1.2.0-py2.7.egg/singularity/registry/main/push.py", line 47, in main
    compress=not args.nocompress)
  File "/home/dave/.local/lib/python2.7/site-packages/singularity-1.2.0-py2.7.egg/singularity/registry/client/push.py", line 118, in push
    'datafile': (upload_to, open(upload_from, 'rb'), 'text/plain')})
  File "build/bdist.linux-x86_64/egg/requests_toolbelt/multipart/encoder.py", line 119, in __init__
  File "build/bdist.linux-x86_64/egg/requests_toolbelt/multipart/encoder.py", line 240, in _prepare_parts
  File "build/bdist.linux-x86_64/egg/requests_toolbelt/multipart/encoder.py", line 488, in from_field
  File "build/bdist.linux-x86_64/egg/requests_toolbelt/multipart/encoder.py", line 466, in coerce_data
  File "build/bdist.linux-x86_64/egg/requests_toolbelt/multipart/encoder.py", line 529, in __init__
  File "build/bdist.linux-x86_64/egg/requests_toolbelt/multipart/encoder.py", line 410, in encode_with
AttributeError: 'dict' object has no attribute 'encode'

what should it be? I'd like something meaningful, customized for the registry, etc.

I found https://github.com/singularityhub/sregistry/blob/master/scripts/prepare_instance.sh to be an interesting mix between Ubuntu specifica and general install instructions with pip and the downloaded Anaconda. Is Anaconda truly required or could we also just work with a regular installable (and deinstallable (!)) system-wide installed python?

The two pip-installed packages do no require the extra pip, for instance https://packages.debian.org/search?keywords=python-ipaddress or https://packages.debian.org/search?keywords=python-oauth2client, so these packages I also expect in Ubuntu. Does this possibly mean that pip does not need to be installed, either?

I can update or backport for you as required.

hey @dctrud are the tags for your biocontainers (originally) containing slashes, and they are replaced with -? I think we would want to either replace with something else, or just ensure there is only one slash (two slashes look weird)

What are your thoughts on how this would work? Feel free to answer as many or as few of the below as you choose. Specifically:

• Do you envision an integration with Singularity Registry, Hub, or both?
• How does a registry endpoint (a local fileystem), or the hub cloud storage (Google Cloud Storage), interact with Globus clients?
• Who has control to initiate a sharing of a container? How does it work?
• Can entire folders / collections be shared automatically? How does this work (in terms of webhooks - does each collection send out a webhook that users subscribe to, or does the registry keep track of subscribed users and initiate the transfer?)
• Do you see benefit to Globus over using Singularity pull?
• Does it complicate things if the Singularity Registry is external to the cluster (since requires docker).
• What level of control is a user given to update images in a Registry?
• Does some file system watcher (e.g., inofity) drive the notifications for changed images?
• Should a share be automatic, or manual? In both cases, how is the model created? Updated?

Please post your thoughts, or disregard the questions above and answer "if I had a way to easily share images it would look like this..."

Thanks!

Docker-compose is not working.

vanessa/sregistry does not exists in docker hub.

The entire sregistry (singularity-python) application should be provided as an image, so the user doesn't need to install it with python / dependencies. Very meta :)

This would be active only for registries with globus_auth activated

http://globus-search-docs.s3-website-us-east-1.amazonaws.com/stable/api/ingest.html

Hi all,

I'm trying to use the development branch of vsoch/singularity-python with sregistry and I'm getting some errors.

I would like to post this directly in the right repository, please @vsoch, could you open the issues feature for this repository? If not, which communication channel do you prefer to post issues about this repository?

I'm running run_client.sh example and I have Python 2.7.12 installed. is Python 3 a requirement?

Lets go with the bugs,

• First, in this line, L86 It crashes because image does not exist.
• Then, Calling to $ sregistry list $IMAGE with an existing image, I get the following error with the date:

...
  File "/usr/local/lib/python2.7/dist-packages/singularity-1.2.0-py2.7.egg/singularity/registry/client/query.py", line 213, in container_search
    datetime_object = datetime.strptime(c['add_date'], '%Y-%m-%dT%H:%M:%S.%fZ')
ValueError: time data '2017-10-13T01:47:50.824098-05:00' does not match format '%Y-%m-%dT%H:%M:%S.%fZ'

• There is another error calling to: $ sregistry delete $IMAGE

This line crashes [L54](https://github.com/vsoch/singularity-python/blob/development/singularity/registry/client/delete.py#L54) . It seems that python 3 `input` command equivalent in python 2.5 is `raw_input` . 
* Finally,  if I try to force-delete `$ sregistry delete --force $IMAGE` It returns:

ERROR Beep boop! Internal Server Error: 500

Thanks in advance!

With docker it is possible to run a local registry as a proxy cache we would like to have something similar for singularity, is this currently possible? (we mostly want to keep the internet traffic down when someone starts 1000 jobs which all start by pulling the same image). As a similar issue we'd quite like to run a local registry, but I can't see how to configure sregistry for purely local use.

Clair is the CoreOS project for security static analysis of containers, scanning them for security issues (from databases of known CVEs). I'd like to propose adding support to sregistry for scanning containers using Clair.

Though Clair is centered around docker or appc images, it has been used to scan openvz templates, which are .tar archives - see FastVPSEestiOu/check_openvz_mirror_with_clair. I'm pretty sure something similar could be done for singularity images.

This is something I'm planning to work on, and thought I'd add a ticket here in case it's of interest to others / there are any thoughts? I'm thinking I will be working to:

• Create a stand alone python tool to scan a singularity image via Clair API
• Investigating contributing this to singularity-python (if it's of interest there)
• Integrate with sregistry - celery jobs for Clair scanning periodically / on push?

Would welcome any input on if this is of interest for sregistry, or more generally.

Hi all,

what do you think about providing an implementation of the authentication against FiWare through Oauth2?

https://www.fiware.org/

Please, feel free to close this issue if you are not interested in.

Thanks in advance!
Víctor.

Hi,

I've setup a local sregistry, and add my user as superuser and admin. I've copy-paste the token from gui, then put in my home .sregistry, then do a push :

sregistry push openmpi.img --name openmpi [================================] 196/196 MB - 00:00:0096 MB - 00:00:00 Upload finished! [Return status 403 Unauthorized]
sregistry list No container collections found.
How can i debug this ?
Thanks

Martin

Dear all,

I'm playing with the sregistry, and I would like to have the registry working for my project. This will expose a bunch of images submitted by users, but some of them could have private data or source code, tetc.

I have seen the PRIVATE_ONLY variable in the config file in config file and #21 . it allow admins to decide if they want to have all collection public or private.

Can users by themselves to decide which images can be public or not?

I would like to have this feature, and the PRIVATE_BY_DEFAULT=yes/no instead of the PRIVATE_ONLY variable.

How it sounds for you?

BR,
Víctor.

This is a shared issue to put custom settings that we might want to add:

• Gzip Compression

Right now, the client does compression at -6. However, if -9 is important for reproducibility, it should be default. However, it comes at the cost of speed - it's rather slow to do. I think if -6 is ok, we should default to that. However if there is question, we should let the user decide with the most reasonable default.

• Colors / Logo

I'd like it to be easy for the registry to (further) customize itself, with an institution logo / accent color.

Hi,

I'm thinking in the possibility of managing several remote registries and how to communicate them.

I would like to transfer images between registries.
E.g:

sregistry push shub://origin/registry shub://destination/registry

Do you think sregistry could support this or this must be managed out of sregistry?

Best regards,
Víctor.

Hi all,

I install a local sregistry web service in a private network and I'm playing with it.

I'm using singlarity-python from this repository: https://github.com/vsoch/singularity-python/ (in particular the development branch) to push an pull images to the registry web service.

Push command works perfectly and I can see the new images registered in the web service, but when I try to locally pull the same image I get an error:

   ...
    image_file = self.download(url=result['image'],
KeyError: 'image'

Diving a little in the code, I was able to explore (check/error) looking for a solution.
I think the following line is wrong:
https://github.com/vsoch/singularity-python/blob/development/singularity/registry/client/pull.py#L51

if we change this line and write the following, it works (at least in my local test):

 image_file = self.download(url=url,

Hope it helps!
BR,
Víctor.

Hi,

I'm trying to follow the registration process here https://singularityhub.github.io/sregistry/setup.html, but it seems that template-registry.md does not exist anywhere.

See https://github.com/singularityhub/sregistry/blob/master/shub/apps/base/management/commands/register.py#L57

Please, Let me know how to proceed!

Thanks

On a registry with a very large number of containers the size map looks like a nice rainbow, but doesn't show much useful info since names can't be seen. Possibly create the map at collection level (sizes of each collection) when there are more than X containers?

Hi all,

I'm accessing to usage section of the sregistry web service and I get the following page:

Here you can see the following help:

sregistry pull victorsndvg/hello-world/master
sregistry pull victorsndvg/hello-world/master --name customname.img
singularity pull shub://10.38.3.117/victorsndvg/hello-world/master
singularity pull --name customname.img shub://10.38.3.117/victorsndvg/hello-world/master

I think the tag must be concatenated after a colon instead a slash. I'm right?

Like this:

sregistry pull victorsndvg/hello-world:master
sregistry pull victorsndvg/hello-world:master --name customname.img
singularity pull shub://10.38.3.117/victorsndvg/hello-world:master
singularity pull --name customname.img shub://10.38.3.117/victorsndvg/hello-world:master

BR,
Víctor.

sregistry client is using gzip with -9 (best) compression. Default for gzip is -6.

-9 is giving minimal improvement in compression over the default, and takes a lot longer. When building and pushing 100s/1000s of images in an automated manner this can become a big issue. E.g. creating Singularity versions of docker biocontainers, sregistry's gzip -9 is the slowest step of the process.

Example compression on image created from the Singularity examples/centos definition. Original image is 769M on disk. Compression with -9 takes >4x as long, saves <1M in final compressed image size:

# Default gzip compression
21:08 $ time gzip centos.img && ls -la centos.img.gz
real    0m27.862s
user    0m27.529s
sys     0m0.332s
-rwxr-xr-x. 1 dave dave 163302185 Aug 26 09:07 centos.img.gz

# Current sregistry (-9) compression
21:10 $ time gzip -9 centos.img && ls -la centos.img.gz
real    2m3.429s
user    2m3.060s
sys     0m0.368s
-rwxr-xr-x. 1 dave dave 162702257 Aug 26 09:07 centos.img.gz

Hi,

after installing the web server and authenticate with social auth, now I want to try to push/pull images using sregistry.

I build singularity-python locally as explained in the Singularity file. Now I've access to sregistry.

How can I use it properly?

First, I get the "no secrets file exists" error. I create a .sregistry at $HOME with "NAME" and "SECRET_KEY" keys. I'm not sure if this is Ok, I don't read anywhere how to build this file.

But then when trying to call to sregistry push command I get:

TypeError: inspect() got an unexpected keyword argument 'deffile'

I'm completely lost.

Thanks,
Víctor

Installed the vsoch/singularity-python development branch to use the sregistry client on CentOS 7 (uses python 2.7.5).

On running there is an import error for JSONDecodeError from json. This is beause json.JSONDecodeError wasn't added until later versions of the json package than are with python 2.7.5.

A workaround would be to use the fact that JSONDecodeError subclasses ValueError, which is what the old json returns (https://stackoverflow.com/a/35214768)

index d5a4a89..61ef45b 100644
--- a/singularity/hub/__init__.py
+++ b/singularity/hub/__init__.py
@@ -25,9 +25,7 @@ SOFTWARE.

 '''

-from simplejson import JSONDecodeError as SimpleJSONDecodeError
 from requests.exceptions import HTTPError
-from json import JSONDecodeError

 from singularity.logger import bot
 import requests
@@ -244,7 +242,7 @@ class ApiConnection(object):
                 try:
                     response =  response.json()

-                except (SimpleJSONDecodeError, JSONDecodeError):
+                except (ValueError):
                     bot.error("The server returned a malformed response.")
                     sys.exit(1)

Hi,

I have a frozen image stored in SRegistry, I try to overwrite this image and SRegistry works as expected. Nothing change on SRegistry side:

$ sregistry push --name alpine/container --tag 3.6 alpine.3.6.simg

Upload finished! [Return status 403 alpine/container:3.6 is frozen, push not allowed.]

But sregistry command return a successful exit code.

Do you think this is the right behaviour?
It could be possible to return an error code in this case?

Let me know your thoughts!

A registry should be able to produce an API endpoint to identify itself.

Am running an sregistry instance at https://sregistry.randomroad.net. In `~/.sregisty' the endpoint is correctly set (from copying out of the token page on the web interface).

 { "token": "...", "username": "...", "base": "https://sregistry.randomroad.net"}

However, the sregistry client is still trying to use 127.0.0.1:

20:44 $ sregistry push --name examples/docker docker.img
Traceback (most recent call last): 1/1 MB - 00:00:00 0/1 MB - 00:00:00
  File "/home/dave/.local/bin/sregistry", line 9, in <module>
    load_entry_point('singularity==1.2.0', 'console_scripts', 'sregistry')()
  File "/home/dave/.local/lib/python2.7/site-packages/singularity-1.2.0-py2.7.egg/singularity/registry/main/__init__.py", line 204, in main
    subparser=subparsers[args.command])
  File "/home/dave/.local/lib/python2.7/site-packages/singularity-1.2.0-py2.7.egg/singularity/registry/main/push.py", line 47, in main
    compress=not args.nocompress)
  File "/home/dave/.local/lib/python2.7/site-packages/singularity-1.2.0-py2.7.egg/singularity/registry/client/push.py", line 126, in push
    r = requests.post(url, data=monitor, headers=headers)
  File "/home/dave/.local/lib/python2.7/site-packages/requests/api.py", line 112, in post
    return request('post', url, data=data, json=json, **kwargs)
  File "/home/dave/.local/lib/python2.7/site-packages/requests/api.py", line 58, in request
    return session.request(method=method, url=url, **kwargs)
  File "/home/dave/.local/lib/python2.7/site-packages/requests/sessions.py", line 508, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/dave/.local/lib/python2.7/site-packages/requests/sessions.py", line 640, in send
    history = [resp for resp in gen] if allow_redirects else []
  File "/home/dave/.local/lib/python2.7/site-packages/requests/sessions.py", line 218, in resolve_redirects
    **adapter_kwargs
  File "/home/dave/.local/lib/python2.7/site-packages/requests/sessions.py", line 618, in send
    r = adapter.send(request, **kwargs)
  File "/home/dave/.local/lib/python2.7/site-packages/requests/adapters.py", line 506, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='127.0.0.1', port=443): Max retries exceeded with url: /api/push/ (Caused by SSLError(CertificateError("hostname '127.0.0.1' doesn't match 'www.randomroad.net'",),))

Looks like the api_base is hard coded in singularity/registry/client/__init__.py and is not being overidden by the setting in ~/.sregistry. By editing the api_base in the source file I can make the client talk to my server.

Hi,

I'm getting into troubles to configure sregistry with https. I try to follow your instructions, but generate_cert.sh referenced in install.md does not exist.

• pull (could also be done with singularity)
• list (should show some subset of containers)
• share - need to think more about this one, but should be possible to get sharing link / etc for container.

Hi,

I don't know if this is the expected behaviour, but HELP_INSTITUTION_SITE, defined in this line, prepends the SRegistry URL (E.g. localhost IP address) in the Request a build section of the main page of the SRegistry.

I think, at least in my case, it is more general to allow to link external web sites.

What do you think?

From a user:

Our authorization for other systems is entirely built around LDAP whenever possible. I think that's quite common for HPC.

InfoSec etc. really want to know how authorization is maintained for different systems - and management at a central LDAP directory using LDAP groups makes that much more straightforward to keep an eye on, and document

The email uniqueness thing goes like this - 'Researcher A works in a lab, but also in a core facility. They only have one email address but 2 separate accounts on the HPC system. This is so that their work/data between core and lab can be separated' - so we have many people with 2 usernames sharing same email - and they use our various systems with the 2 different usernames for different work that needs to be separated.

If would like to expose the sregistry API behind an api management layer. How would I delegate authorization requests to an external authority. Currently services running behind the gateways are passed a JWT or other standard header with the user identity and request context. How would I go about this with sregistry?

This is harder to do with a web based application, but we will need to have tests for confidence that the application continues to work.

So sorry... I don't mean to nitpick. On the front page:
Let's get familiar with the Registry, see Introduction Let's get started.
This would read better as:

Let's get familiar with the Registry - so get started by looking at the Introduction

Hi all,

I think how Singularity-hub works together with GitHub is really great and easy for end-users.

As GitLab is the "community GitHub clone", I think it could be interesting to provide a bridge to connect local installations of SRegistry and GitLab. This brings the power to custom site administrators to provide fast/easy building of Singularity containers.

What do you think?