add option to not build all branches about singularityhub.github.io

[vsoch]

hey @HenrikBengtsson ! Ok, I added the functions and views, give it a spin! I think this is you:

https://singularity-hub.org/collections/44/branches/edit

[vsoch]

And a few quick feedback points to the poster!

• the option of disabling builds for a branch is a great idea. Currently, Singularity Hub doesn't do anything to read the repo, only the ping from Github, so the strategy to store the metadata in a file with the repo doesn't (at least immediately) plug in nicely. I think we could take one of two approaches:

1. disable all other branch builds by default, make the user add it in the web interface if a new one is desired to be built
2. enable all branches by default, make the user disable it in the web interface if a new one is desired.

If git tags is built into Github (and just a means for Github to look up a commit id) I definitely see this being an option for the shub:// endpoint, and this would be a two part issue (to also bring up on the main Singularity software page) involving the following:

• adding functionality in singularity hub to provide an API endpoint that translates a tag uri to a commit --> image.
• adding the endpoint option to the Singularity command line tool to parse.

[HenrikBengtsson]

Approach 1, where all branches but master is disabled by default, appeals to me. I keeps the learning curve at a minimum while still providing the option to build other branches too.

I take your word for it, but I'm not sure I understand why it would be more complicated to have additional metadata in a file that lives in the repos. I assume (=guessing) the repos is cloned to the Hub server already now, isn't it? Then, although suboptimal, before launching the build, it could check for an optional .singularityhub.yml file and the based on the content decide on building / not building.

The GitHub API supports listing tags, e.g.

$ curl https://api.github.com/repos/hb-slash/busybox/tags
[
  {
    "name": "1.0.1",
    "zipball_url": "https://api.github.com/repos/hb-slash/busybox/zipball/1.0.1",
    "tarball_url": "https://api.github.com/repos/hb-slash/busybox/tarball/1.0.1",
    "commit": {
      "sha": "5dee487e546f74e746061fe5c72f1d22b5568173",
      "url": "https://api.github.com/repos/hb-slash/busybox/commits/5dee487e546f74e746061fe5c72f1d22b5568173"
    }
  },
  {
    "name": "1.0.0",
    "zipball_url": "https://api.github.com/repos/hb-slash/busybox/zipball/1.0.0",
    "tarball_url": "https://api.github.com/repos/hb-slash/busybox/tarball/1.0.0",
    "commit": {
      "sha": "3ef96e2a3657f0c30878ed8a46380d171b0d4cba",
      "url": "https://api.github.com/repos/hb-slash/busybox/commits/3ef96e2a3657f0c30878ed8a46380d171b0d4cba"
    }
  }
]

[vsoch]

Nothing is actually cloned to the server, all of that business is done with a builder, which is a separate server (there are many) deployed on demand that clones the repo, builds the image, and sends it back. Ideally we would want to prevent the build from entering the queue period, meaning that server should never get deployed (and then check that there is a spec file that says not to build, haha).

The tags are super cool! I think my one hesitation is wondering how many people actually use tags. I use Github a ton, and most of my repos look like this:

  curl https://api.github.com/repos/vsoch/singularity-scientific-example/tags
  [

  ]

It's like a sandwich with empty bread, haha.
Ok cool, I'll give a go at implementing a function so the user must ask specifically to enable a branch. It should be pretty reasonable actually - a ping will be checked for the branch, and if it is master (or in the list of enabled) will go through. Otherwise, it will just let it go, or even better, store it to give the user the option to enable it later. I'll have to add clear docs so that users aren't confused by why a branch isn't building.

[HenrikBengtsson]

I see.

Git tags are quite widely used in my part of the Git / GitHub world, but it's also used in your neighborhood ;)

curl -s https://api.github.com/repos/singularityware/singularity/tags | grep name
    "name": "2.2",
    "name": "2.1.2",
    "name": "2.1.1",
    "name": "2.1",
    "name": "2.0",
    "name": "1.0",

Also, if you don't know about Git Flow, have a look at:

• how it works: http://jeffkreeftmeijer.com/2010/why-arent-you-using-git-flow/
• git plugin: https://github.com/petervanderdoes/gitflow-avh

If you do your own home-cooked braching / release models, as I did for a long time, this fits almost everything you need. I can't imagine not working without this branching / release model.

Basic summary:

• Never commit to master.
• I always have master reflect the most recent stable release, and do all development in develop. That way a random user will not install my development code by mistake.
• Always commit to the develop branch (or below feature branches); git checkout develop.
• For a new release, git flow release start 1.2.0 (creates temporary branch release/1.2.0), update version, CHANGELOG, and do final tests, then git commit and finally git flow release finish 1.2.0. This will merge release/1.2.0 into develop and master, tag the top master commit 1.2.0 and finally remove release/1.2.0 again. The git push and git push --tags and you've made the new release public.
• If you're adding new features requiring major / long-term work, do git flow feature start new-stuff, which will branch the develop branching into feature/new-stuff. When well tested etc, do git flow feature finish new-stuff to merge it into develop and remove that feature branch.

[HenrikBengtsson]

(Yes, that's my "slash" organization.)

Beautiful. Thxs. I've tried it out and it seems to work, e.g. I've deleted the previous feature/yoo build, enabled it in the hub settings (*), pushed new commits to GitHub, and, yep, the feature/yoo branch was rebuilt.

I see that you're also providing access to tags via refs/tags/<tag>. That's great. I understand that support for @<tag> and @<branch> would require an update to Singularity. Personally, I think that would be somewhat cleaner references, especially if tags are used for versions, e.g. hb-slash/[email protected]. Also, in addition to GitHub using it to reference commits, this @ format also fits what other projects uses, e.g. R and Spack.

(*) The first time I forgot to 'Submit' (kept two browser windows open). Maybe those on/off switches could be made to save changes automatically?

[vsoch]

Branch building is now activated by the user on demand, closing issue. Thanks!

[qobilidop]

May I ask what's the current status of supporting tags? I'm a beginner user and I might be wrong. Seems it's not on singularity hub yet?

[vsoch]

Do you mean singularity hub building specific tags of Github repos? Right now we treat a tag as a branch, so for example a Singularity file in repo vsoch/containers and branch development would have uri 'vsoch/containers:development'

[qobilidop]

I mean the git tags discussed above. Now I see that refs/tags/<tag> is already provided. Sorry for the confusing question.