Manually trigger builds about singularityhub.github.io

[vsoch]

I'll definitely give this a thinking over - I agree that it's not optimal to have less power to determine triggers. A lot of users will likely be building from a docker image first, and so they would want to have those two things working nicely together. We could definitely go the manual "click to trigger way," but we also should look at having webhooks come from Docker Hub. We also might have users that want continuous integration services (eg, circle, travis) to trigger builds from somewhere given tests pass.

[vsoch]

@TormodLandet - if this would be an option for a local registry, would this satisfy this need? If so, I'll add the issue to the milestone for version 2.0 of singularity hub, and if not I can think more about this.

[TormodLandet]

I think it would be best to have it available both places, but feel free to down-prioritize this, it is not critical, but just quite nice to have :-)

[vsoch]

I agree. I'll see what I can do ::)

[apeltzer]

Bumping on this - this would be indeed a great feature. I don't always update the main project with my Singularity file in it, as it pulls in other dependencies when building the container image, so manually triggering these would be best for my use case. I think I could live with both (local registry vs yours), but I think its a good feature for both cases!

[vsoch]

hey @apeltzer I've been thinking over this, let me describe what I have in mind, and if it would fit with your workflow!

Singularity Registry

A registry would be optimized for building locally, with only occasional pushes to Singularity Hub to make a latest image available for users outside the institution. This is important to note because the registry will have a trigger command that comes directly from the command line. How is Github involved then?
There are several ideas, and here is the one that I (currently) like best:

• registry is registered in Singularity hub, this creates a Registry database object, associated with one or more users (the cluster admins)
• the admins then log in, and select a Github repository to connect to their registry. This github repo is the one that they have created for their registry.
• when the institution builds, the build is on their resource.
• a successful build sends this trigger to singularity hub, the trigger means a token/refresh token approach. Then Singularity Hub:
  • updates some production branch that it maintains for the registry
  • launches builds for each updated image

The distinction between a registry and a user is that for a registry, one folder in a Github repo is a collection. For a user, one entire Github repo is a collection. This is mostly done to encourage researchers to work in a more modular / reproducible way, although people will definitely argue with me about that one. Now for the typical user use cases:

Singularity Hub

In the current model, a user must commit to Github to build. Branches must be manually activated. This isn't ideal if you want more of a controlled trigger. What I don't want to do is give any user the same control to build as a registry - for the user, having the runscript / image tied in a knot nicely with a Github commit is essential. A manual trigger doesn't ensure that the runscript is nicely associated with a particular commit that can again be found.

What I do think might fit the bill, however, is to give users more ways to trigger a build, and specifically, ways that still have some requirement of capturing the metadata. For example, Singularity Hub would offer all of the following for a single collection

1. build from Github commits (as is now)
2. build based on a webhook from Docker Hub (this would fit in to what I think your use case is, doing testing first and then building after it's done on Docker Hub)

I was thinking about as a supplement to 2, it could also be the case that a user connecting a Docker Hub repo means forgetting about looking for the repo / Singularity file, and just taking the docker image and pulling it, and providing it as a Singularity image. This doesn't seem so logical to me because you would get the same thing with one easy line:

singularity pull shub://dockername/reponame
singularity pull docker://dockername/reponame

3. The webhook could also come from the continuous integration service, for example circleci. It's also pretty common that users start at Github from here.

For all of the webhooks, we would need a way to validate them. Github is great because there is a hashed secret sent that I verify, but the others don't seem to have that, so it would likely need to be based on other header fields and the host they came from. Could you describe better your workflow, and what kind of triggers you had in mind? We should figure out something that meets the following:

• fits in nicely with workflows that are easy / common (eg, Github, Docker Hub, CircleCI), and this coincides with taking advantage of these third party services
• supports reproducibility in that each build has a definite link to the build specification file, so a contained could be linked back to it.

[apeltzer]

Currently my developmental model is as follows:

I work on updated applications, test them (real data, simulations and also some JUnit tests) and then would like to update the images. The current situation does not allow me to trigger a build after my tests - its just able to trigger a build whenever I commit to one of my repositories. Ideally I could have a webhook in Jenkins CI, that is then triggering a build (Docker Hub has something like this), so whenever I decide to produce a release after testing, I can trigger that webhook and a new image is built.

[vsoch]

cool! So a circle/travis webhook would fit the bill?

[apeltzer]

I'm using Jenkins CI, but I guess this is also possible to integrate with. In the worst case I could simply set up a shell command and trigger manually with curl or something.

I think that should work yes!

[vsoch]

hey @apeltzer - do you know if Jenkins is able to send a deployment status back to GIthub on successful build? If so, I could use this webhook -->https://developer.github.com/v3/activity/events/types/#deploymentstatusevent This would actually be a much more extendable solution than implementing each service separately, since (I think?) they all might be sending back the statuses. The concern I have with the endpoints as single units is that they don't send any kind of hash/key to validate the payload, which means that if someone found the endpoint, they could be malicious and trigger builds of your images. If we use Github, we get the verification.

[vsoch]

ok this will need testing once I get it in a cloud, but now there is a setting that will change the build to check for the deployment status (instead of the commit):

[apeltzer]

Looks good to me - I'll check how Jenkins can integrate with this, but am really sure it can do that. I know that there is full github support anyways .

[vsoch]

hey @TormodLandet ! I'm adding support for multiple recipe files in a repo, and I'm curious to know how you would want this to work with a "Trigger Build" button. If you look at issue #49 it makes sense to rebuild a container, because we've already discovered a particular build recipe. In the case of triggering manually, we would not have this information. So either it would default to something, and/or ask the user to input the build recipe path. Eg: subfolder/Singularity.ubuntu would be in a folder "subfolder" at the root of the repository, building an image with tag "ubuntu." I'd like to add this feature for you if we can figure out this detail! Right now I've just enforced that the user needs to disable automatic building before a manual trigger is possible. For automatic builds, I am allowing a re-build to be done, with the same rules as per a normal queue in terms of quota, etc.

[TormodLandet]

I think that having a list of recipes already discovered with a "Trigger build" button next to each of them would be sufficient. I guess you would like to have a list of recipes found in the repo anyway?

For my use case I am happy with turning off automatic builds and only pressing the "Trigger build" button once I have pushed the underlying docker files to docker hub and manually verified that they work properly

[vsoch]

hey @TormodLandet, that's a good idea, but it would take a little more processing oomph on the main application server. The recipe names are discovered from the webhook pings as changed files, and so even if we saved a changed list there would be some potential misalignment with our list and what is actually in the repo. It also gets messy quickly because we have to keep track of different branches. We want to avoid any kind of heavy duty parsing for the server, and have this handled by the builders. By the time a builder is granted a new job, he should already know the repo, recipe, branch, commit, etc.

So, a few options we could try:

• when manual trigger is asked for, retrieve the file contents of some branch and pre-populate Singularity recipes: https://developer.github.com/v3/git/trees/
• force the user to specify a path, branch, manually. If it doesn't exist, the builder will just error and exit.
• attempt parsing all known recipes from the webhooks, if there is some misalignment, it will exit and report back anyway.
• something else?

I think more ideal would be hooking to the Docker Hub API, but they don't have good checks for the validity of webhooks. Another option is to have the collection build given Deployment status, but this comes from Github and you wouldn't yet have had chance to manually test the Docker images.