GithubHelp home page GithubHelp logo

pachyderm / openshift-operator Goto Github PK

View Code? Open in Web Editor NEW
3.0 25.0 6.0 47.68 MB

Operator for deploying Pachyderm on OpenShift

License: Apache License 2.0

Dockerfile 1.53% Makefile 5.40% Go 91.13% Shell 1.93%

openshift-operator's Introduction

Pachyderm Operator

The pachyderm operator is an application responsible for installing, watching and managing Pachyderm resources in your Openshift cluster.

Pachyderm is the data foundation for machine learning. Pachyderm provides industry leading data versioning, pipelines and lineage that allow data science teams to automate the machine learning lifecycle and optimize their machine learning operations (MLOps).

Usage

1. Using AWS S3 for Pachd storage

  • Create a secret which contains the AWS S3 storage information
$ oc create secret generic pachyderm-aws --from-literal access-id=ABCDEFGHIJKLMNOPQR --from-literal access-secret=dkhfjdshfj/fjkdshfiuUjmfhdsjkhfjdhs/KLhdfuiseh --from-literal bucket=pachyderm-bucket --from-literal region=us-east-1`

secret/pachyderm-aws created
$
  • Create a Pachyderm custom resource in the same namespace
$ cat <<EOF> pachyderm-cr.yaml
apiVersion: aiml.pachyderm.com/v1beta1
kind: Pachyderm
metadata:
  name: pachyderm-sample
  namespace: pachyderm-test
spec:
  console:
    disable: true
  pachd:
    metrics:
      disable: false
    storage:
      amazon:
        credentialSecretName: pachyderm-aws
      backend: AMAZON
EOF
$ oc create -f pachyderm-cr.yaml
pachyderm.aiml.pachyderm.com/pachyderm-sample created
$ 
  • Ensure pachyderm is up and running
$ oc get pachyderm pachyderm-sample -o yaml | yq e '.status' -
phase: Running
$   

2. User-provided postgresql database

  • Set postgresql to disabled in pachyderm.spec.postgresql

  • Provide postgresql instance information in pachyderm.spec.pachd.postgresql

  • Create a k8s secret to hold the postgresql password. It should have a key postgres-password

openshift-operator's People

Contributors

chainlink avatar jooho avatar nadegepepin avatar ochienged avatar opdevbot avatar

Stargazers

 avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

openshift-operator's Issues

Validate Pach Version + Add to sample

If I try and deploy the sample in the current repo, I get an index out of bounds exception. This can be traced to the new pachyderm version code, we should present a better error message here if the pach version is not specified in the given CRD instance. We should also add Pachyderm version to config/samples/aiml_v1beta1_pachyderm.yaml

Support encryption of credentials on Pachyderm CR

In some user environments, custom resources that represent deployments may be maintained as files in github or other git repositories. The Pachyderm CR includes credentials for S3 buckets, which would need to either be stored separately in a kubernetes Secret, or be stored on the Pachyderm CR in an encrypted form, so that the s3 credentials are not exposed on git repositories.

Allow images to be overridden in CRD

We have inputs in the CRD for image repository / tag, but we don't use those values if they are specified in the CRD (instead using the values in images.json) I believe the ordering should be

  1. Use image and tag values from the crd if specified
  2. Fall back on the images.json values

Handle empty values in values.yaml template

Values in the values.yaml should only be passed through if they are specified in the given CRD. That way they can rely on chart defaults. As an example, the chart fails to apply by default as reverse gets set to empty string, which is not a valid boolean value

Error message:

values don't meet the specifications of the schema(s) in the following chart(s):\npachyderm:\n- pachd.storage.amazon.reverse: Invalid type. Expected: boolean, given: string\n

This is because the value is rendered out as

reverse: <nil>

https://github.com/pachyderm/openshift-operator/blob/main/hack/charts/2.0.3/values.yaml#L297

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.