For further information about this repository, refer to the corresponding Medium article.
Tested with:
- Python: 3.8.16
- Docker: 20.10.22
- Kind 0.17.0
For information refer to the official documentation.
To summarize the steps:
kind create cluster
export PIPELINE_VERSION=1.8.5
kubectl apply -k "github.com/kubeflow/pipelines/manifests/kustomize/cluster-scoped-resources?ref=$PIPELINE_VERSION"
kubectl wait --for condition=established --timeout=60s crd/applications.app.k8s.io
kubectl apply -k "github.com/kubeflow/pipelines/manifests/kustomize/env/platform-agnostic-pns?ref=$PIPELINE_VERSION"
# Wait a couple of minutes.
kubectl config set-context --current --namespace=kubeflow
# Port forwading for the UI
kubectl port-forward -n kubeflow svc/ml-pipeline-ui 8080:80
Kubeflow UI will be at http://localhost:8080.
Local environment is set up by using make tool.
make
The task train_model
involves component specification and docker registry. For further details refer to the official documentation or the Medium article.
kubectl -n kubeflow create secret docker-registry registry-secret \
--docker-server=https://index.docker.io/v1/ \
--docker-username=<username> \
--docker-password=<access-key> \
--docker-email=<email>
source venv/bin/activate
cd custom_components/train_model_component
kfp components build . --component-filepattern train_model.py
docker tag test_kubeflow_train_model:latest <username>/kubeflow:latest
docker push <username>/kubeflow:latest
Make sure that the component metadata image name is correct, otherwise update it with the tag used to push the image into the Docker registry in the previous step.
source venv/bin/activate
python pipeline.py
kind delete cluster
Pull requests are welcome. For major changes, please open an issue first to discuss what you would like to change.