GithubHelp home page GithubHelp logo

kfserving-operator's Introduction

KFServing Operator

Overview

This charm encompasses the Kubernetes Python operator for KFServing (see CharmHub).

The KFServing operator is a Python script that wrap the latest released version of KFServing, providing lifecycle management and handling events such as install, upgrade, integrate, and remove.

Install

To install KFServing, run:

juju deploy kfserving

For more information, see https://juju.is/docs

kfserving-operator's People

Contributors

ca-scribner avatar dnplas avatar knkski avatar miaaltieri avatar

Watchers

avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar

Forkers

miaaltieri

kfserving-operator's Issues

[ERROR] if kind is a CRD, it should be installed before calling Start

Brief description

Full Charmed Kubeflow bundle have been deploy. Need for KFServing. Kubeflow is working just right, but we encounter an error when deploying the KFServing charm.

Environment

  • Ubuntu 20.04
  • Self-managed Kubernetes cluster
  • Charmed Kubeflow v1.4 Full bundle charm (installed via juju)

What has been performed

  1. Install Full Charmed Kubeflow Bundle (it works correctly)
  2. Deploy KFserving charm

What Happened
The following error is obtained after running

juju deploy kfserving

Created resources status is:

pod/kfserving-5cccd745ff-84rn6                   0/1     CrashLoopBackOff   9          22m
pod/kfserving-operator-0                         1/1     Running            0          22m
service/kfserving                            ClusterIP      10.111.141.246   <none>          8080/TCP,9443/TCP                                                                                                                                                         22m
service/kfserving-operator                   ClusterIP      10.102.27.82     <none>          30666/TCP                                                                                                                                                                 22m
service/kfserving-webhook-server-service     ClusterIP      10.102.168.241   <none>          443/TCP                                                                                                                                                                   22m
deployment.apps/kfserving                   0/1     1            0           22m
replicaset.apps/kfserving-5cccd745ff                   1         1         0       22m
statefulset.apps/kfserving-operator                   1/1     22m

Logs of kfserving pod's container

{"level":"info","ts":1649672394.0135996,"logger":"entrypoint","msg":"Setting up client for manager"}
{"level":"info","ts":1649672394.0140657,"logger":"entrypoint","msg":"Setting up manager"}
I0411 10:19:55.064208       1 request.go:645] Throttling request took 1.035076668s, request: GET:https://10.96.0.1:443/apis/autoscaling/v2beta1?timeout=32s
{"level":"info","ts":1649672396.1216981,"logger":"controller-runtime.metrics","msg":"metrics server is starting to listen","addr":"127.0.0.1:8080"}
{"level":"info","ts":1649672396.1220202,"logger":"entrypoint","msg":"Registering Components."}
{"level":"info","ts":1649672396.1220503,"logger":"entrypoint","msg":"Setting up KFServing v1alpha1 scheme"}
{"level":"info","ts":1649672396.122212,"logger":"entrypoint","msg":"Setting up KFServing v1alpha2 scheme"}
{"level":"info","ts":1649672396.1223402,"logger":"entrypoint","msg":"Setting up KFServing v1beta1 scheme"}
{"level":"info","ts":1649672396.1224382,"logger":"entrypoint","msg":"Setting up Knative scheme"}
{"level":"info","ts":1649672396.1227343,"logger":"entrypoint","msg":"Setting up Istio schemes"}
{"level":"info","ts":1649672396.1229227,"logger":"entrypoint","msg":"Setting up core scheme"}
{"level":"info","ts":1649672396.1230392,"logger":"setup","msg":"Setting up v1beta1 controller"}
{"level":"info","ts":1649672396.1241481,"logger":"setup","msg":"Setting up v1beta1 TrainedModel controller"}
{"level":"info","ts":1649672396.1242077,"logger":"entrypoint","msg":"setting up webhook server"}
{"level":"info","ts":1649672396.1242206,"logger":"entrypoint","msg":"registering webhooks to the webhook server"}
{"level":"info","ts":1649672396.1242962,"logger":"controller-runtime.webhook","msg":"registering webhook","path":"/mutate-pods"}
{"level":"info","ts":1649672396.1243567,"logger":"controller-runtime.builder","msg":"skip registering a mutating webhook, admission.Defaulter interface is not implemented","GVK":"serving.kubeflow.org/v1alpha1, Kind=TrainedModel"}
{"level":"info","ts":1649672396.1243808,"logger":"controller-runtime.builder","msg":"Registering a validating webhook","GVK":"serving.kubeflow.org/v1alpha1, Kind=TrainedModel","path":"/validate-serving-kubeflow-org-v1alpha1-trainedmodel"}
{"level":"info","ts":1649672396.1244278,"logger":"controller-runtime.webhook","msg":"registering webhook","path":"/validate-serving-kubeflow-org-v1alpha1-trainedmodel"}
{"level":"info","ts":1649672396.1245108,"logger":"controller-runtime.builder","msg":"Registering a mutating webhook","GVK":"serving.kubeflow.org/v1alpha2, Kind=InferenceService","path":"/mutate-serving-kubeflow-org-v1alpha2-inferenceservice"}
{"level":"info","ts":1649672396.1245544,"logger":"controller-runtime.webhook","msg":"registering webhook","path":"/mutate-serving-kubeflow-org-v1alpha2-inferenceservice"}
{"level":"info","ts":1649672396.1246002,"logger":"controller-runtime.builder","msg":"Registering a validating webhook","GVK":"serving.kubeflow.org/v1alpha2, Kind=InferenceService","path":"/validate-serving-kubeflow-org-v1alpha2-inferenceservice"}
{"level":"info","ts":1649672396.1246467,"logger":"controller-runtime.webhook","msg":"registering webhook","path":"/validate-serving-kubeflow-org-v1alpha2-inferenceservice"}
{"level":"info","ts":1649672396.1247542,"logger":"controller-runtime.webhook","msg":"registering webhook","path":"/convert"}
{"level":"info","ts":1649672396.1247866,"logger":"controller-runtime.builder","msg":"conversion webhook enabled","object":{"name":""}}
{"level":"info","ts":1649672396.1248136,"logger":"controller-runtime.builder","msg":"Registering a mutating webhook","GVK":"serving.kubeflow.org/v1beta1, Kind=InferenceService","path":"/mutate-serving-kubeflow-org-v1beta1-inferenceservice"}
{"level":"info","ts":1649672396.1248405,"logger":"controller-runtime.webhook","msg":"registering webhook","path":"/mutate-serving-kubeflow-org-v1beta1-inferenceservice"}
{"level":"info","ts":1649672396.1248696,"logger":"controller-runtime.builder","msg":"Registering a validating webhook","GVK":"serving.kubeflow.org/v1beta1, Kind=InferenceService","path":"/validate-serving-kubeflow-org-v1beta1-inferenceservice"}
{"level":"info","ts":1649672396.1249049,"logger":"controller-runtime.webhook","msg":"registering webhook","path":"/validate-serving-kubeflow-org-v1beta1-inferenceservice"}
{"level":"info","ts":1649672396.1249723,"logger":"controller-runtime.builder","msg":"conversion webhook enabled","object":{"name":""}}
{"level":"info","ts":1649672396.1249862,"logger":"entrypoint","msg":"Starting the Cmd."}
{"level":"info","ts":1649672396.125226,"logger":"controller-runtime.webhook.webhooks","msg":"starting webhook server"}
{"level":"info","ts":1649672396.1252196,"logger":"controller-runtime.manager","msg":"starting metrics server","path":"/metrics"}
{"level":"info","ts":1649672396.1253963,"logger":"controller-runtime.manager.controller.inferenceservice","msg":"Starting EventSource","reconciler group":"serving.kubeflow.org","reconciler kind":"InferenceService","source":"kind source: /, Kind="}
{"level":"info","ts":1649672396.1253529,"logger":"controller-runtime.manager.controller.trainedmodel","msg":"Starting EventSource","reconciler group":"serving.kubeflow.org","reconciler kind":"TrainedModel","source":"kind source: /, Kind="}
{"level":"info","ts":1649672396.1258228,"logger":"controller-runtime.certwatcher","msg":"Updated current TLS certificate"}
{"level":"info","ts":1649672396.126057,"logger":"controller-runtime.certwatcher","msg":"Starting certificate watcher"}
{"level":"info","ts":1649672396.126253,"logger":"controller-runtime.webhook","msg":"serving webhook server","host":"","port":9443}
{"level":"info","ts":1649672396.2258558,"logger":"controller-runtime.manager.controller.inferenceservice","msg":"Starting EventSource","reconciler group":"serving.kubeflow.org","reconciler kind":"InferenceService","source":"kind source: /, Kind="}
{"level":"info","ts":1649672396.226206,"logger":"controller-runtime.manager.controller.trainedmodel","msg":"Starting Controller","reconciler group":"serving.kubeflow.org","reconciler kind":"TrainedModel"}
{"level":"error","ts":1649672399.7161593,"logger":"controller-runtime.source","msg":"if kind is a CRD, it should be installed before calling Start","kind":"Service.serving.knative.dev","error":"no matches for kind \"Service\" in version \"serving.knative.dev/v1\"","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/pkg/mod/github.com/go-logr/[email protected]/zapr.go:132\nsigs.k8s.io/controller-runtime/pkg/source.(*Kind).Start\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/source/source.go:117\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:159\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:205\nsigs.k8s.io/controller-runtime/pkg/manager.(*controllerManager).startRunnable.func1\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/manager/internal.go:691"}
{"level":"info","ts":1649672399.7162907,"logger":"controller-runtime.manager.controller.trainedmodel","msg":"Starting workers","reconciler group":"serving.kubeflow.org","reconciler kind":"TrainedModel","worker count":1}
{"level":"info","ts":1649672399.7164187,"logger":"controller-runtime.manager.controller.trainedmodel","msg":"Stopping workers","reconciler group":"serving.kubeflow.org","reconciler kind":"TrainedModel"}
{"level":"info","ts":1649672399.7164598,"logger":"controller-runtime.webhook","msg":"shutting down webhook server"}
{"level":"error","ts":1649672399.7165453,"logger":"entrypoint","msg":"unable to run the manager","error":"no matches for kind \"Service\" in version \"serving.knative.dev/v1\"","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/pkg/mod/github.com/go-logr/[email protected]/zapr.go:132\nmain.main\n\t/go/src/github.com/kubeflow/kfserving/cmd/manager/main.go:183\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:203"}

After juju deploy kfservingtwo crd are added.
Listing kfserving CRDs kubectl get crd | grep serving outputs:

inferenceservices.serving.kubeflow.org                2022-04-11T10:02:59Z
trainedmodels.serving.kubeflow.org                    2022-04-11T10:02:59Z

By describing them they seem to be okey kubectl describe crd <CRDs-serving-name>outputs:

Status:
  Accepted Names:
    Kind:       TrainedModel
    List Kind:  TrainedModelList
    Plural:     trainedmodels
    Short Names:
      tm
    Singular:  trainedmodel
  Conditions:
    Last Transition Time:  2022-04-11T10:02:59Z
    Message:               no conflicts found
    Reason:                NoConflicts
    Status:                True
    Type:                  NamesAccepted
    Last Transition Time:  2022-04-11T10:02:59Z
    Message:               the initial names have been accepted
    Reason:                InitialNamesAccepted
    Status:                True
    Type:                  Established
    Last Transition Time:  2022-04-11T10:02:59Z
    Message:               spec.preserveUnknownFields: Invalid value: true: must be false
    Reason:                Violations
    Status:                True
    Type:                  NonStructuralSchema
  Stored Versions:
    v1alpha1
Events:  <none>

HELP

We are not sure why this is happening as we have not install any CRD outside the ones included in Kubeflow charm. Anyone has experienced this situation?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.