skai-x / elastic-jupyter-operator Goto Github PK

View Code? Open in Web Editor NEW

192.0 192.0 28.0 1.26 MB

Cloud-native way to provide elastic Jupyter Notebooks on Kubernetes. Run remote kernels, natively.

License: Apache License 2.0

Dockerfile 0.88% Makefile 4.10% Go 94.86% Shell 0.15%

cloud-native jupyter jupyter-kernels jupyter-notebook kubernetes

elastic-jupyter-operator's People

Contributors

Stargazers

Watchers

elastic-jupyter-operator's Issues

[feat] Image warm start using JupyterKernelSpec

[feat] Support Launching Notebooks with CRD Directly

Running the custom kernel example - error when starting kernel from JupyterLab

When going through the custom kernel example everything starts correctly in k8s, but when starting the kernel from the notebook it produced a 500 HTTP error:

Seems like the error is some version incompatibility with enterprise gateway. Here are the logs from the gateway:

Error: TypeError: __init__() got an unexpected keyword argument 'kernel_id'

 HTTPServerRequest(protocol='http', host='jupytergateway-elastic-with-custom-kernels.default:8888', method='POST', uri='/api/kernels', version='HTTP/1.1', remote_ip='172.17.0.1')
Traceback (most recent call last):
File "/opt/conda/lib/python3.9/site-packages/tornado/web.py", line 1704, in _execute
result = await result
File "/opt/conda/lib/python3.9/site-packages/enterprise_gateway/services/kernels/handlers.py", line 90, in post
await super(MainKernelHandler, self).post()
File "/opt/conda/lib/python3.9/site-packages/jupyter_server/services/kernels/handlers.py", line 47, in post
kernel_id = await km.start_kernel(kernel_name=model["name"], path=model.get("path"))
File "/opt/conda/lib/python3.9/site-packages/enterprise_gateway/services/kernels/remotemanager.py", line 156, in start_kernel
kernel_id = await super(RemoteMappingKernelManager, self).start_kernel(*args, **kwargs)
File "/opt/conda/lib/python3.9/site-packages/jupyter_server/services/kernels/kernelmanager.py", line 211, in start_kernel
kernel_id = await ensure_async(self.pinned_superclass.start_kernel(self, **kwargs))
File "/opt/conda/lib/python3.9/site-packages/jupyter_server/utils.py", line 189, in ensure_async
result = await obj
File "/opt/conda/lib/python3.9/site-packages/jupyter_client/multikernelmanager.py", line 194, in _async_start_kernel
await fut
File "/opt/conda/lib/python3.9/site-packages/jupyter_client/multikernelmanager.py", line 165, in _add_kernel_when_ready
await kernel_awaitable
File "/opt/conda/lib/python3.9/site-packages/jupyter_client/utils.py", line 34, in ensure_async
return await obj
File "/opt/conda/lib/python3.9/site-packages/enterprise_gateway/services/kernels/remotemanager.py", line 365, in start_kernel
await super(RemoteKernelManager, self).start_kernel(**kwargs)
File "/opt/conda/lib/python3.9/site-packages/jupyter_client/manager.py", line 362, in _async_start_kernel
raise e
File "/opt/conda/lib/python3.9/site-packages/jupyter_client/manager.py", line 351, in _async_start_kernel
await ensure_async(self._launch_kernel(kernel_cmd, **kw))
File "/opt/conda/lib/python3.9/site-packages/jupyter_client/utils.py", line 34, in ensure_async
return await obj
File "/opt/conda/lib/python3.9/site-packages/enterprise_gateway/services/kernels/remotemanager.py", line 425, in _launch_kernel
proxy = await self.process_proxy.launch_process(kernel_cmd, **kwargs)
File "/opt/conda/lib/python3.9/site-packages/enterprise_gateway/services/processproxies/kubeflow.py", line 49, in launch_process
await super(KubeflowProcessProxy, self).launch_process(kernel_cmd, **kwargs)
File "/opt/conda/lib/python3.9/site-packages/enterprise_gateway/services/processproxies/container.py", line 81, in launch_process
self.local_proc = self.launch_kernel(kernel_cmd, **kwargs)
File "/opt/conda/lib/python3.9/site-packages/enterprise_gateway/services/processproxies/processproxy.py", line 456, in launch_kernel
return launch_kernel(cmd, **kwargs)
File "/opt/conda/lib/python3.9/site-packages/jupyter_client/launcher.py", line 169, in launch_kernel
raise ex
File "/opt/conda/lib/python3.9/site-packages/jupyter_client/launcher.py", line 157, in launch_kernel
proc = Popen(cmd, **kwargs)
TypeError: __init__() got an unexpected keyword argument 'kernel_id'
[E 220201 17:06:22 web:2239] 500 POST /api/kernels (172.17.0.1) 23.14ms

This seems related to this issue: jupyter-server/enterprise_gateway#1024

Seems like it's using jupyter-client 7.1.0, which might be the problem? I'll try to confirm

[bug] Kernel completed when run

Traceback (most recent call last):
File "/usr/local/bin/kernel-launchers/python/scripts/launch_ipykernel.py", line 289, in _validate_port_range
lower_port = int(port_ranges[0])
ValueError: invalid literal for int() with base 10: 'None'

Followed quickstart, but no kernels, did I miss something?

Hey folks,

Firstly - thank you so much for starting this project! It looks really promising, very excited to try it out.

Disclaimer: I'm not super familiar with enterprise gateway, so I might just be doing something stupid (likely).

I ran make deploy, then followed the quickstart guide. Everything worked, but when I went to localhost:8888 I get an empty notebook with no kernels....

Was I supposed to do something else? I feel like I'm just missing a step.

Thanks for pointing me in the right direction!

Matthew

[feat] Set the kernel owner reference to gateway

feat(notebook): Avoid token in the URL

It is hard to get the token without getting the logs, we should figure out if there's a way to avoid tokens.

chore: Support CI with GitHub Actions

chore: Merge enterprise-gateway namespace with elastic-jupyter-operator-system

Now we get two ns, it is not necessary.

bug: Kernel image does not work in the JupyterKernelSpec

[feat] Use 2.5.0 enterprise gateway

feat(dashboard): Support Events/Logs

feat(vscode): Investigate if we can support vscode-ssh with containers on Kubernetes

docs: Update README to illustrate the new design

[docs] Add roadmap

feat(dashboard): Support basic operations like CRUD (mainly focus on get/list and create) resources

bug(arm64) Cannot run the operator

Hi, do you know how to resolve this issue?

eugene@Eugenes-Mac-mini elastic-jupyter-operator % kubectl port-forward deploy/jupyternotebook-elastic 8888:8888
Error from server (NotFound): deployments.apps "jupyternotebook-elastic" not found

All steps

eugene@Eugenes-Mac-mini elastic-jupyter-operator % minikube start 
😄  minikube v1.24.0 on Darwin 12.1 (arm64)
✨  Automatically selected the docker driver
👍  Starting control plane node minikube in cluster minikube
🚜  Pulling base image ...
🔥  Creating docker container (CPUs=2, Memory=4000MB) ...
🐳  Preparing Kubernetes v1.22.3 on Docker 20.10.8 ...
    ▪ Generating certificates and keys ...
    ▪ Booting up control plane ...
    ▪ Configuring RBAC rules ...
🔎  Verifying Kubernetes components...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟  Enabled addons: storage-provisioner, default-storageclass
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default


eugene@Eugenes-Mac-mini elastic-jupyter-operator % kubectl apply -f ./hack/enterprise_gateway/prepare.yaml
namespace/enterprise-gateway created
serviceaccount/enterprise-gateway-sa created
clusterrole.rbac.authorization.k8s.io/enterprise-gateway-controller created
clusterrole.rbac.authorization.k8s.io/kernel-controller created
clusterrolebinding.rbac.authorization.k8s.io/enterprise-gateway-controller created

eugene@Eugenes-Mac-mini elastic-jupyter-operator % make deploy

go: creating new go.mod: module tmp
go get: installing executables with 'go get' in module mode is deprecated.
  To adjust and download dependencies of the current module, use 'go get -d'.
  To install using requirements of the current module, use 'go install'.
  To install ignoring the current module, use 'go install' with a version,
  like 'go install example.com/cmd@latest'.
  For more information, see https://golang.org/doc/go-get-install-deprecation
  or run 'go help get' or 'go help install'.
go get: added github.com/fatih/color v1.9.0
go get: added github.com/go-logr/logr v0.2.0
go get: added github.com/gobuffalo/flect v0.2.2
go get: added github.com/gogo/protobuf v1.3.1
go get: added github.com/google/gofuzz v1.1.0
go get: added github.com/inconshreveable/mousetrap v1.0.0
go get: added github.com/json-iterator/go v1.1.10
go get: added github.com/mattn/go-colorable v0.1.8
go get: added github.com/mattn/go-isatty v0.0.12
go get: added github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd
go get: added github.com/modern-go/reflect2 v1.0.1
go get: added github.com/spf13/cobra v1.1.1
go get: added github.com/spf13/pflag v1.0.5
go get: added golang.org/x/mod v0.3.0
go get: added golang.org/x/net v0.0.0-20201110031124-69a78807bb2b
go get: added golang.org/x/sys v0.0.0-20201112073958-5cba982894dd
go get: added golang.org/x/text v0.3.4
go get: added golang.org/x/tools v0.0.0-20200616133436-c1934b75d054
go get: added golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1
go get: added gopkg.in/inf.v0 v0.9.1
go get: added gopkg.in/yaml.v2 v2.3.0
go get: added gopkg.in/yaml.v3 v3.0.0-20200615113413-eeeca48fe776
go get: added k8s.io/api v0.20.2
go get: added k8s.io/apiextensions-apiserver v0.20.2
go get: added k8s.io/apimachinery v0.20.2
go get: added k8s.io/klog/v2 v2.4.0
go get: added k8s.io/utils v0.0.0-20201110183641-67b214c5f920
go get: added sigs.k8s.io/controller-tools v0.5.0
go get: added sigs.k8s.io/structured-merge-diff/v4 v4.0.2
go get: added sigs.k8s.io/yaml v1.2.0
/Users/eugene/go/bin/controller-gen "crd:trivialVersions=true" rbac:roleName=manager-role webhook paths="./..." output:crd:artifacts:config=config/crd/bases
go: creating new go.mod: module tmp
go get: installing executables with 'go get' in module mode is deprecated.
  To adjust and download dependencies of the current module, use 'go get -d'.
  To install using requirements of the current module, use 'go install'.
  To install ignoring the current module, use 'go install' with a version,
  like 'go install example.com/cmd@latest'.
  For more information, see https://golang.org/doc/go-get-install-deprecation
  or run 'go help get' or 'go help install'.
go get: added cloud.google.com/go v0.38.0
go get: added github.com/Azure/go-ansiterm v0.0.0-20170929234023-d6e3b3328b78
go get: added github.com/Azure/go-autorest/autorest v0.9.0
go get: added github.com/Azure/go-autorest/autorest/adal v0.5.0
go get: added github.com/Azure/go-autorest/autorest/date v0.1.0
go get: added github.com/Azure/go-autorest/logger v0.1.0
go get: added github.com/Azure/go-autorest/tracing v0.5.0
go get: added github.com/MakeNowJust/heredoc v0.0.0-20170808103936-bb23615498cd
go get: added github.com/PuerkitoBio/purell v1.1.1
go get: added github.com/PuerkitoBio/urlesc v0.0.0-20170810143723-de5bf2ad4578
go get: added github.com/chai2010/gettext-go v0.0.0-20160711120539-c6fed771bfd5
go get: added github.com/davecgh/go-spew v1.1.1
go get: added github.com/dgrijalva/jwt-go v3.2.0+incompatible
go get: added github.com/docker/docker v0.7.3-0.20190327010347-be7ac8be2ae0
go get: added github.com/docker/spdystream v0.0.0-20160310174837-449fdfce4d96
go get: added github.com/emicklei/go-restful v2.9.5+incompatible
go get: added github.com/evanphx/json-patch v4.5.0+incompatible
go get: added github.com/exponent-io/jsonpath v0.0.0-20151013193312-d6023ce2651d
go get: added github.com/ghodss/yaml v1.0.0
go get: added github.com/go-errors/errors v1.0.1
go get: added github.com/go-openapi/jsonpointer v0.19.3
go get: added github.com/go-openapi/jsonreference v0.19.3
go get: added github.com/go-openapi/spec v0.19.5
go get: added github.com/go-openapi/swag v0.19.5
go get: added github.com/gogo/protobuf v1.2.2-0.20190723190241-65acae22fc9d
go get: added github.com/golang/protobuf v1.3.2
go get: added github.com/google/btree v1.0.0
go get: added github.com/google/go-cmp v0.3.0
go get: added github.com/google/gofuzz v1.0.0
go get: added github.com/google/shlex v0.0.0-20191202100458-e7afc7fbc510
go get: added github.com/googleapis/gnostic v0.0.0-20170729233727-0c5108395e2d
go get: added github.com/gophercloud/gophercloud v0.1.0
go get: added github.com/gregjones/httpcache v0.0.0-20180305231024-9cad4c3443a7
go get: added github.com/hashicorp/errwrap v1.0.0
go get: added github.com/hashicorp/go-multierror v1.0.0
go get: added github.com/hashicorp/golang-lru v0.5.1
go get: added github.com/imdario/mergo v0.3.5
go get: added github.com/inconshreveable/mousetrap v1.0.0
go get: added github.com/jonboulle/clockwork v0.1.0
go get: added github.com/json-iterator/go v1.1.8
go get: added github.com/konsorten/go-windows-terminal-sequences v1.0.1
go get: added github.com/liggitt/tabwriter v0.0.0-20181228230101-89fcab3d43de
go get: added github.com/mailru/easyjson v0.7.0
go get: added github.com/mattn/go-runewidth v0.0.7
go get: added github.com/mitchellh/go-wordwrap v1.0.0
go get: added github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd
go get: added github.com/modern-go/reflect2 v1.0.1
go get: added github.com/olekukonko/tablewriter v0.0.4
go get: added github.com/peterbourgon/diskv v2.0.1+incompatible
go get: added github.com/pkg/errors v0.8.1
go get: added github.com/posener/complete/v2 v2.0.1-alpha.12
go get: added github.com/posener/script v1.0.4
go get: added github.com/russross/blackfriday v1.5.2
go get: added github.com/sirupsen/logrus v1.4.2
go get: added github.com/spf13/cobra v0.0.5
go get: added github.com/spf13/pflag v1.0.5
go get: added github.com/xlab/treeprint v0.0.0-20181112141820-a009c3971eca
go get: added golang.org/x/crypto v0.0.0-20190923035154-9ee001bba392
go get: added golang.org/x/net v0.0.0-20191004110552-13f9640d40b9
go get: added golang.org/x/oauth2 v0.0.0-20190604053449-0f29369cfe45
go get: added golang.org/x/sys v0.0.0-20190922100055-0a153f010e69
go get: added golang.org/x/text v0.3.2
go get: added golang.org/x/time v0.0.0-20190308202827-9d24e82272b4
go get: added google.golang.org/appengine v1.5.0
go get: added gopkg.in/inf.v0 v0.9.1
go get: added gopkg.in/yaml.v2 v2.2.4
go get: added gopkg.in/yaml.v3 v3.0.0-20191026110619-0b21df46bc1d
go get: added k8s.io/api v0.17.0
go get: added k8s.io/apimachinery v0.17.0
go get: added k8s.io/cli-runtime v0.17.0
go get: added k8s.io/client-go v0.17.0
go get: added k8s.io/component-base v0.17.0
go get: added k8s.io/klog v1.0.0
go get: added k8s.io/kube-openapi v0.0.0-20191107075043-30be4d16710a
go get: added k8s.io/kubectl v0.0.0-20191219154910-1528d4eea6dd
go get: added k8s.io/utils v0.0.0-20191114184206-e782cd3c129f
go get: added sigs.k8s.io/kustomize v2.0.3+incompatible
go get: added sigs.k8s.io/kustomize/api v0.3.2
go get: added sigs.k8s.io/kustomize/cmd/config v0.0.5
go get: added sigs.k8s.io/kustomize/cmd/kubectl v0.0.3
go get: added sigs.k8s.io/kustomize/kustomize/v3 v3.5.4
go get: added sigs.k8s.io/kustomize/kyaml v0.0.6
go get: added sigs.k8s.io/yaml v1.1.0
cd config/manager && /Users/eugene/go/bin/kustomize edit set image controller=ccr.ccs.tencentyun.com/kubeflow-oteam/elastic-jupyter-operator:latest
/Users/eugene/go/bin/kustomize build config/default | kubectl apply -f -
namespace/elastic-jupyter-operator-system created
customresourcedefinition.apiextensions.k8s.io/jupytergateways.kubeflow.tkestack.io created
customresourcedefinition.apiextensions.k8s.io/jupyterkernels.kubeflow.tkestack.io created
customresourcedefinition.apiextensions.k8s.io/jupyterkernelspecs.kubeflow.tkestack.io created
customresourcedefinition.apiextensions.k8s.io/jupyterkerneltemplates.kubeflow.tkestack.io created
customresourcedefinition.apiextensions.k8s.io/jupyternotebooks.kubeflow.tkestack.io created
role.rbac.authorization.k8s.io/elastic-jupyter-operator-leader-election-role created
clusterrole.rbac.authorization.k8s.io/elastic-jupyter-operator-manager-role created
rolebinding.rbac.authorization.k8s.io/elastic-jupyter-operator-leader-election-rolebinding created
clusterrolebinding.rbac.authorization.k8s.io/elastic-jupyter-operator-manager-rolebinding created
deployment.apps/elastic-jupyter-operator-controller-manager created

eugene@Eugenes-Mac-mini elastic-jupyter-operator % kubectl apply -f ./examples/elastic/kubeflow.tkestack.io_v1alpha1_jupyternotebook.yaml
jupyternotebook.kubeflow.tkestack.io/jupyternotebook-elastic created


eugene@Eugenes-Mac-mini elastic-jupyter-operator % kubectl apply -f ./examples/elastic/kubeflow.tkestack.io_v1alpha1_jupytergateway.yaml
jupytergateway.kubeflow.tkestack.io/jupytergateway-elastic created


eugene@Eugenes-Mac-mini elastic-jupyter-operator % kubectl port-forward deploy/jupyternotebook-elastic 8888:8888
Error from server (NotFound): deployments.apps "jupyternotebook-elastic" not found

[feat] Design Kernel CRD to ease the maintanance

bug(upstream): The metadata is empty in podTemplate

kubernetes-sigs/controller-tools#448

could support informer,lister and clientset?

hi there,
could anybody support informer,lister and clientset? it's inconvenient to handle the CR. thanks a ton!

-BR

Customized Labels & Annotations are not supported in notebook CRD refered pod & deployment

Behavior:
kubectl apply the following yaml:

apiVersion: kubeflow.tkestack.io/v1alpha1
kind: JupyterNotebook
metadata:
  name: jupyternotebook-elastic-with-custom-kernels
  namespace: public
spec:
  gateway:
    name: jupytergateway-elastic-with-custom-kernels
    namespace: public
  # Disable the password and token based auth in this example,
  # please do not do it in PROD.
  auth:
    mode: disable
  template:
    metadata:
      annotations:
        cni: macvlan

get the following:

apiVersion: v1
items:
- apiVersion: kubeflow.tkestack.io/v1alpha1
  kind: JupyterNotebook
  metadata:
    annotations:
      cni: macvlan
    creationTimestamp: "2022-03-21T09:50:31Z"
    generation: 3
    name: jupyternotebook-elastic
    namespace: public
    resourceVersion: "152659678"
    selfLink: /apis/kubeflow.tkestack.io/v1alpha1/namespaces/public/jupyternotebooks/jupyternotebook-elastic
    uid: 12d437c1-45e5-4afa-99bf-38371533046e
  spec:
    auth:
      mode: disable
    gateway:
      name: jupytergateway-elastic
      namespace: public
    template:
      metadata: {}
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

Manually Editing the jupyternotebook to add spec.template.metadata.annotations does not work either.

Found that spec.template.metadata.annotations is never used/considered in reconciling deloyment:

elastic-jupyter-operator/pkg/notebook/reconcile.go

Line 71 in 4415a90

desired, err := r.gen.DesiredDeploymentWithoutOwner()

elastic-jupyter-operator/pkg/notebook/generate.go

Line 116 in 0d1472b

ObjectMeta: metav1.ObjectMeta{

elastic-jupyter-operator/pkg/notebook/generate.go

Line 125 in 0d1472b

ObjectMeta: metav1.ObjectMeta{

How should I run or deploy this operator?

The deployment steps I understand are as follows:

Install this project - the elastic-jupyter-operator project itself is an operator
kubectl apply -f ./hack/enterprise_gateway/prepare.yaml
After the above two steps, the crd has been created, and CR is deployed next.
JupyterNotebook CR: kubectl apply -f ./examples/elastic/kubeflow.tkestack.io_v1alpha1_jupyternotebook.yaml
JupyterGateway CR: kubectl apply -f ./examples/elastic/kubeflow.tkestack.io_v1alpha1_jupytergateway.yaml

Is there any problem with the above steps?
Then during my local execution, a problem occurred in step 1: when executing make deploy:

admin@Will elastic-jupyter-operator % make deploy
/Users/admin/go/bin/controller-gen "crd:trivialVersions=true" rbac:roleName=manager-role webhook paths="./..." output:crd:artifacts:config=config/crd/bases
cd config/manager && /Users/admin/go/bin/kustomize edit set image controller=ghcr.io/skai-x/elastic-jupyter-operator:latest
/Users/admin/go/bin/kustomize build config/default | kubectl apply -f -
Unable to connect to the server: dial tcp 10.40.3.250:6443: i/o timeout
make: *** [deploy] Error 1

How should I proceed?
My biggest difficulty right now is: How to execute the elastic-jupyter-operator project so that it generates 5 necessary CRDs?
The above problems are encountered when installing this project!
OR, Is there a corresponding helm chart?

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.