jvujjini / kfp-tekton

Kubeflow Pipeline with Tekton backend. Enables SDK functionalities (Compile, Upload, Launch) and KFP UI with execution and artifact tracking

Kubeflow Pipelines with Tekton

Project to bring Kubeflow Pipelines and Tekton together. The project is split in phases and driven according to this design doc. The current code allows you run Kubeflow Pipelines with Tekton backend end to end.

• Create your Pipeline using Kubeflow Pipelines DSL, and compile it to Tekton YAML.
• Upload the compiled Tekton YAML to KFP engine (API and UI), and run end to end with logging and artifacts tracking enabled.

Tekton

The Tekton Pipelines project provides Kubernetes-style resources for declaring CI/CD-style pipelines. Tekton introduces several new CRDs including Task, Pipeline, TaskRun, and PipelineRun. A PipelineRun represents a single running instance of a Pipeline and is responsible for creating a Pod for each of its Tasks and as many containers within each Pod as it has Steps. Please look for more details in Tekton repo.

Kubeflow Pipeline with Tekton backend

We are currently using Kubeflow Pipelines 0.5.1 and Tekton 0.14.0 for this project.

Installing Kubeflow Pipelines with Tekton backend

Getting started with KFP Tekton deployment

Getting Started

Getting started with KFP Tekton SDK

Developer Guide

Developer Guide

Available Features and Implementation Details

Available Features and Implementation Details

Compiler Status Report

Compilation Tests Status Report

Samples

Samples being run end to end for verification

KFP, Argo and Tekton Features Comparison

KFP, Argo and Tekton Features Comparison

Design Doc

Design Doc

CD Foundation

• CD Foundation MLOps Sig.
• Instructions to join

Additional Reference Materials: KFP and TFX

• Kubeflow and TFX Pipelines
• Kubeflow and TFX Pipelines talk at Tensorflow World