infinitelooper / vertex-pipelines-ci-cd-intro

Vertex AI Pipelines CI/CD Introduction

An introduction to CI/CD workflow on Vertex AI through example.

Pre-requiste (optional)

This builds upon the official Google codelab: Intro to Vertex Pipelines by re-using the same code to demonostrate a example MLOps level 2 CI/CD pipeline automation workflow.

It is not required and recommended to complete that excercise first so as to become familar with how to run a Vertex Pipeline manually.

Setup

Enable APIs

1. In the GCP Cloud Shell, run the commands below. 

export PROJECT=$DEVSHELL_PROJECT_ID

gcloud services enable aiplatform.googleapis.com --project $PROJECT_ID
gcloud services enable artifactregistry.googleapis.com --project $PROJECT_ID
gcloud services enable bigquerystorage.googleapis.com --project $PROJECT_ID
gcloud services enable cloudbuild.googleapis.com --project $PROJECT_ID
gcloud services enable clouddeploy.googleapis.com --project $PROJECT_ID
gcloud services enable container.googleapis.com --project $PROJECT_ID
gcloud services enable containerregistry.googleapis.com --project $PROJECT_ID
gcloud services enable monitoring.googleapis.com --project $PROJECT_ID
gcloud services enable notebooks.googleapis.com --project $PROJECT_ID
gcloud services enable run.googleapis.com --project $PROJECT_ID

Create Workbench Notebook Instance

2. Navigate to Vertex Workbench User Managed Notebooks and create a python notebook instance (or use the cloud shell command below) 2.1. At the top of the screen, click "NEW NOTEBOOK" 2.2. Use the first option for a notebook "Python 3" 2.3. For the Region, select the first option "us-central1"  2.4. Click "Create"

   gcloud notebooks instances create vertex-ai-pipelines-cicd-intro \
       --vm-image-project=deeplearning-platform-release \
       --vm-image-family=common-cpu-notebooks \
       --machine-type=n1-standard-4 \
       --location=us-central1-a

3. Once the notebook instance is created, clone this repository via the GUI or terminal:

   git clone https://github.com/justinjm/vertex-pipelines-ci-cd-intro.git

Optional You can also work from the Cloud Shell Editor and click the button below to clone and open this repository in your own Cloud Shell instance:

3.a Complete workbench notebook instance setup by installing dependencies and restarting kernel

See notebook 00_setup.ipynb

Create Cloud Storage Bucket

4. Create a cloud storage bucket

create bucket and save the name for use later

BUCKET_NAME=gs://$GOOGLE_CLOUD_PROJECT-aip-pipeline-cicd
gsutil mb -l us-central1 $BUCKET_NAME

Now give our compute service account access to this bucket. This will ensure that Vertex Pipelines has the necessary permissions to write files to this bucket. Run the following command to add this permission:

gcloud projects describe $GOOGLE_CLOUD_PROJECT > project-info.txt
PROJECT_NUM=$(cat project-info.txt | sed -nre 's:.*projectNumber\: (.*):\1:p')
SVC_ACCOUNT="${PROJECT_NUM//\'/}[email protected]"
gcloud projects add-iam-policy-binding $GOOGLE_CLOUD_PROJECT --member serviceAccount:$SVC_ACCOUNT --role roles/storage.objectAdmin

Create Artifact Registry

6. Create artifact registry

See notebook 02_image_build.ipynb

7. Setup Cloud Build Trigger

7.a. Create cloudbuild.yaml

See cloudbuild.yaml for pre-built one, update with your constants as needed

7.b Setup GitHub Build trigger

Setup a Cloud Build trigger to execute the vertex pipeline execution whenever a commit is made to the main branch in the repository

First, Connect to a GitHub repository

Then, setup a cloud build trigger from the connected GitHub repo

Building repositories from GitHub  |  Cloud Build Documentation  |  Google Cloud

Console:

• name: automl-beans
• region: us-central1
• description: Trigger for implementing CI/CD workflow for Vertex Pipelines
• event: "push to a new branch"
• source:
  • repository: select repository from dropdown
  • branch: ^main$
• Configuration: Autodetected

Grant Vertex AI access to Cloud Build

8. Grant permissions to vertex AI service agents

Grant the following roles to the Cloud Build service account {PROJECT-NUMBER}@cloudbuild.gserviceaccount.com:

• Vertex AI Service Agent (roles/aiplatform.serviceAgent)

You can do this in the console or via gclound:

Console:

1. Goto IAM & Admin page, select your project from the top dropdown
2. Check the box on the right: "Include Google-provided role grants"
3. Find Cloud Build default service account {PROJECT-NUMBER}@cloudbuild.gserviceaccount.com
4. Add role Vertex AI Service Agent (roles/aiplatform.serviceAgent)

gcloud:

gcloud projects describe $GOOGLE_CLOUD_PROJECT > project-info.txt
PROJECT_NUM=$(cat project-info.txt | sed -nre 's:.*projectNumber\: (.*):\1:p')
SVC_ACCOUNT="${PROJECT_NUM//\'/}@cloudbuild.gserviceaccount.com"
gcloud projects add-iam-policy-binding $GOOGLE_CLOUD_PROJECT --member serviceAccount:$SVC_ACCOUNT --role roles/aiplatform.serviceAgent

Resources

• IAM permissions  |  Vertex AI  |  Google Cloud
• Access control with IAM  |  Vertex AI  |  Google Cloud
• Troubleshooting Vertex AI  |  Google Cloud

Workflow

General workflow is as follows:

1. Setup GCP environment:
   • README.md (this document) - setup GCP environment
   • 00_setup.ipynb - setup notebook instabce
   • 02_image_build.ipynb - create artifact registry
2. write code to train and deploy pipeline
   • 01_pipeline_build.ipynb code here and generates file src/pipelines/train_pipeline.py to be used in pipeline
   • src/build_and_deploy.py - main function for building and deploying Vertex Pipeline
   • src/components/classification_eval_model_v2.py - custom component for model evaluation
   • src/component_specs/classification_eval_model_v2.yaml - create based on src/components/classification_eval_model_v2.py
3. create Dockerfile and build image (to containerize for CI/CD)
   • 02_image_build.ipynb
4. manually run pipeline to test
   • 03_pipeline_run.ipynb
5. commit/push code to the linked repository
6. Pipeline job submitted to Vertex AI

Resources

• GCP official Notebook examples
• Codelabs:
  • Intro to Vertex Pipelines
  • Using Vertex ML Metadata with Pipelines
• Schedule pipeline execution with Cloud Scheduler
• Python reference
• vertex-ai-labs/vertex_ai_pipelines_r_model.ipynb at main · RajeshThallam/vertex-ai-labs - example with R
• Vertex Pipelines Setup
  • Configure your Google Cloud project for Vertex AI Pipelines
  • Use a custom service account  |  Vertex AI  |  Google Cloud