thoth-station / jupyternb-build-pipeline

Openshift-pipelines and tekton based pipeline for packaging jupyternb in to image.

jupyternb-build-pipeline

OpenShift-pipeline and TektonCD based pipeline for packaging jupyter notebooks into image for AICoE.

A pipeline that packages Jupyter notebooks into a container image. Container images can be circulated with others in a more convenient way.
The final image will be uploaded to aicoe-notebooks repo after the pipeline is finished.
This registry is Red Hat internal and all images in aicoe-notebooks repo are public by default.

Required repository Structure:

This pipeline looks for a "notebooks" directory in the root of your repo and copies the contents to the Jupyter home directory (/opt/app-root/src/).
The dependencies for notebooks should be managed using a Pipfile and a Pipfile.lock, and these dependencies will be installed during the build time.

We have a template repo which you can use to structure your repo.

Pipeline Setup:

On the GitLab/GitHub repo include the following webhook and also include sesheta as a collaborator.

Webhook: http://thoth-ci.thoth.ultrahook.com
Webhook Secret: *******(contact thoth-station)

Gitlab private/personal repository requirement: Add @sesheta as a collaborator in the project.

Pipeline Details:

The pipeline listens to Tag release events in the repository it is linked to.
On each Tag release, it will trigger a new build and push the updated image to the upshift registry based on the name of the repository.

Note: The name of the repository is to be lowercase.

The pipeline uses an s2i-minimal-notebook(Python3.6) image as the base image for s2i build.
On image build, it pushes the tag to image-registry and also updates the latest tag for the image on the registry.

How to contribute

We welcome contributions, The following components can be worked on:

• Tasks:

  • All new tasks are to be added in the tasks directory.
  • Make sure to add new resource required for the tasks in the resource.yaml

• Pipeline:

  • New pipeline are to be added in the pipeline directory.

• Events:

  • Please create new events for eventlistener, along with the triggertemplate and triggerbindings.

Want to step up an instance

• Setup Tekton Pipeline and Tekton Trigger in cluster.

• Deploy the Triggers, Pipeline, Tasks.

• If behind the VPN, one time setup components:

  • Ultrahook: ultrahook passes the public internet request to services behind VPN

    • ultrahook secret and deployment with destination as jupyternb listener

jupyternb-build-pipeline's Issues

Evaluate migration of images from Upshift central repository to some other location.

Follow up to an email thread

We may need to consider migration of our produced images from the internal Upshift repository to somewhere else. Mind that the produced images may contain sensitive internal data, so they can't land on public Quay repos.

Reference

Hi All,
Thanks for your input so far.  I have added responses in red below.

Maneesh Mehra (CVP):  Thanks for bringing this to the CVP team's attention. We currently use this registry to store our sanity tests image. I have opened https://projects.engineering.redhat.com/browse/CVP-1184 for the CVP team to work on this migration. Do you want to give us a sense of urgency on this ?
Ideally by the end of September 2020, and definitely by the end of December 2020.  

Mike Bonnet (ic3):  We have internal images that are not suitable for public repos. Is there a process for requesting private quay.io repos? Is there guidance about creating a new org, or using an existing one? Note that the current "redhat" org is no longer available for general use.
We have some notes/guidelines that we can update and then point you to.

Mark Eastman (QE):  Pavel may have more details than I but we use these systems to test software before it is released so I am not sure if we can push the images to quay.io during the test phases.
We will look to see if there are other registries at Red Hat that are not in Production, that folks could use.

Regards,
Jeff

Hi Teams,
We are having challenges maintaining the centralized registry with our current capacity.  This central registry at docker-registry.upshift.redhat.com is not a production service, and has low/no SLA.  Going forward, we recommend using quay.io for critical workflows.  If using quay.io is not possible, can you tell us why it is not possible for your team?  In addition, if using quay.io is not possible, what would you suggest as requirements for an 'internal' central registry?

Thanks,
Jeff

Compatibility with gitmodules

As a user of jupyternb build pipeline, I would like to create images from repositories that have gitmodules

An example of such a project is categorical-encoding-internal

Issue pushing images to upshift image registry.

Image are failing to push to upshift registry:

error pushing image "insights-configuration-files-analysis" to "docker://docker-registry.upshift.redhat.com/aicoe-notebooks/insights-configuration-files-analysis:latest": error copying layers and metadata from "containers-storage:[overlay@/var/lib/containers/storage+/var/run/containers/storage:overlay.imagestore=/var/lib/shared,overlay.mount_program=/usr/bin/fuse-overlayfs,overlay.mountopt=nodev,fsync=0]localhost/insights-configuration-files-analysis:latest" to "docker://docker-registry.upshift.redhat.com/aicoe-notebooks/insights-configuration-files-analysis:latest": Error trying to reuse blob sha256:76e525e1b16d0a56473d529790d3f1696078799abf41357103298a09f744f0fe at destination: failed to read from destination repository aicoe-notebooks/insights-configuration-files-analysis: 500 (Internal Server Error)

Add a build pipeline instance for aicoe-aiops/categorical-encoding pushing to Quay.io

Related: aicoe-aiops/categorical-encoding#2

Hi, we need help setting up a new pipeline for building an image from analytics repo, that would package it using s2i-custom-notebook base image and push it to public Quay.io. This image can't lend on the internal registry because it's public and has to be deployable as an example for Operate First - from anywhere, including outside RH firewall.

Can you please help us with that?