rekcurd / dashboard

Project for managing ML model and deploying ML module. It can deploy the Rekcurd service to Kubernetes cluster.

Rekcurd-dashboard

Rekcurd dashboard is the project for managing ML model and deploying ML module. Any Rekcurd service is manageable. It can deploy the Rekcurd service to Kubernetes cluster and can control traffic weights which Istio manages.

Parent Project

https://github.com/rekcurd/community

Components

• Rekcurd: Project for serving ML module.
• Rekcurd-dashboard: Project for managing ML model and deploying ML module.
• Rekcurd-client: Project for integrating ML module.

Screen shot

Environment

• Python 3.6
• flask 1.0.x
• Node.js 8.x
• yarn

Installation

From source:

$ git clone --recursive https://github.com/rekcurd/dashboard.git
$ cd dashboard/frontend
$ yarn install && yarn run build && cd ..
$ pip install -e .

From PyPi directly:

$ pip install rekcurd-dashboard

How to use

Check the belows in detail.

• Backend
• Frontend

Generate settings

You can generate the template of settings.yml file.

$ rekcurd_dashboard template

Boot command

$ rekcurd_dashboard db --settings settings.yml init
$ rekcurd_dashboard db --settings settings.yml migrate
$ rekcurd_dashboard server --settings settings.yml

docker-compose (Deprecated)

# For dev
$ docker-compose -f docker-compose/docker-compose.develop.yaml up
# For prod
$ docker-compose -f docker-compose/docker-compose.production.yaml up

For AWS users

If you run this on AWS (such as EKS), you need to configure aws-cli setting.
Follow the official document.

Rekcurd-dashboard docker container will mount the configuration files,
so the IAM account used by configuration needs to have enough permissions to access to Kubernetes resources on AWS.

# For dev
$ docker-compose -f docker-compose/aws/docker-compose.develop.yaml up
# For prod
$ docker-compose -f docker-compose/aws/docker-compose.production.yaml up

How to use WebUI

See docs in detail.

Test

Unittest

$ python -m unittest test/*/test_*

e2e test

## sudo sh scripts/kube-init.sh
$ sudo sh e2e_test/startup.sh
$ python -m unittest
$ sudo sh e2e_test/cleanup.sh

Kubernetes support

Rekcurd can be run on Kubernetes. See community repository.

dashboard's Issues

Add BBS

Motivation

Bulletin board will be useful to discuss the performance of ML model.

Goal

• Implement BBS

Implementation

• TBD

Add visualization for ML model performance

Motivation

To evaluate the performance of ML model, we need a functionality of evaluate and visualize the ML model.

Goal

• Implement performance visualization

Implementation

• TBD

Stronger security in later version

Motivation

Add stronger security regarding secret when later K8s supports it.

Goal

Encrypt secret data like following
https://kubernetes.io/docs/tasks/administer-cluster/encrypt-data/

Ref: #94

Improve error handling

Motivation

Error handling should be added at right places with clear error messages.

Goal

• Have reasonable error messages that help debug and unit test

Implementation

• Add more error handling at predictable places

Deployable to the application which we don't have a permission

Motivation

Bug.
We can deploy new service to the application which we don't have the write permission.

Goal

• Fix

Implementation

• TBD

Support Dockerfile build and push automatically

Motivation

Reference
https://github.com/GoogleCloudPlatform/knative-build-tutorials/blob/master/docker-build/README.md

Goal

• TBD

Implementation

• TBD

Boot on Kubernetes

Motivation

Monitor and manage drucker-dashboard by Kubernetes.

Goal

• Create configuration file to boot on Kubernetes

Implementation

• Create configuration file to boot on Kubernetes
  • Deployment
  • Service
  • Ingress
  • (Option) Autoscale
  • (Option) https support

Store each evaluation metrics as RDB field

Motivation

Currently, evaluation result is stored as JSON string in result field in EvaluationResult model.
This makes it hard to control data by the metrics such as sorting EvaluationResult by precision.

Goal

Store each evaluation metrics as RDB field to make it easy to control data by the metrics

Implementation

TBD (precison, recall, fvalue can be list, so need to find a good solution to store them)

Add `kubernetes mode flag` to Project settings

Motivation

When Project is created, we can choose the flag of kubernetes mode or not.

Replace library redux-form to formik

Add access control

Motivation

To avoid an accidentally uploading/switching/removing model, we need to implement access control.

Goal

• Implement access control

Implementation

• TBD

Add custom manifest when deploying

Motivation

In some use-cases, user want to create a custom object such as "custom controller".
We need to add options for this case.

Goal

• Add text/file field to write yaml.

Implementation

• feature
• testing

Consider to use FastAPI?

Motivation

https://fastapi.tiangolo.com/

Goal

• write the goal

write the description

Implementation

• feature
• feature
• resource
• testing

Edit (patch) application info

Motivation

Add UI to edit application info.

Goal

• Add UI to edit application info

Implementation

• Add UI to edit application info

Dumping configuration is too slow when we mount a network storage

Motivation

Dumping configuration is too slow when we mount a network storage.
Currently dashboard dumps files when we run "Sync", "Add Application/Server" and so on...

Goal

• Add "Dump Application Configurations" button
• Create batch script to dump configurations

Implementation

• Add "Dump Application Configurations" button
• Create batch script to dump configurations
• Add Unit testing

Use Secret for private git repository

Motivation

For GitHub Enterprise.

Goal

• Use Secret to store SSH key

Implementation

• Add useSecret field to models/application table.
• Allow 1 secret per 1 application.
  • Register the followings.
    • SSH key (id_rsa)
    • private git hostname (config)
    • private git username (config)
    • private git port number (config)
  • Mount Secret to /root/.ssh/.
  • Store id_rsa and config files with mode=600
• testing

config

Host {{ hostname }}
  User {{ username }}
  Port {{ port number }}
  HostName {{ hostname }}
  IdentityFile /root/.ssh/id_rsa
  StrictHostKeyChecking no

id_rsa

-----BEGIN RSA PRIVATE KEY-----
{{ SSH key }}
-----END RSA PRIVATE KEY-----

Reference

• https://kubernetes.io/docs/concepts/configuration/secret/

Edit (patch) project info

Motivation

Add UI to edit project info.

Goal

• Add UI to edit project info

Implementation

• Add UI to edit project info

Add Kubernetes config editor

Motivation

Currently we need to upload the file of Kubernetes token config file.
To extend UX, we add the functionality of editor for Kubernetes token.

Goal

• Implement Kubernetes config editor

Implementation

• TBD

Add access right request function

Motivation

Need an access right request to application/Kubernetes

Support to deploy GCP/AWS/Rancher/kubeadm env

Motivation

There are varieties of Kubernetes environment.
We need to support each environment.

Goal

• Add Select box when deploying new application/service.

Add Select box instead Non Kubernetes/Kubernetes.

Implementation

• TBD

Test coverage up to 0.95

Motivation

Add unittest.
All pull requests must to cover codes up to 0.95.

Goal

• Up to 0.95 test coverage

Implementation

• testing

GPU support

Motivation

GPU support.
https://kubernetes.io/docs/tasks/manage-gpus/scheduling-gpus/

Implementation

• Implement GPU request/limits UI
• Implement nodeSelector set UI
• testing

Cannot work `kube.datadir` if we change the value after registering kubernetes service

Motivation

Cannot work kube.datadir if we change the value after registering kubernetes service. This is because DB entry of kubernetes.config_file is registered using kube.datadir value at the time.

Goal

• Change registration procedure.
• Change loading procedure.

Application permission can be reset if we delete Kubernetes and register it again

Motivation

Application permission can be reset if we delete Kubernetes and register it again

Goal

• TBD

Implementation

• TBD

Add image driven boot option

Motivation

Currently we need to specify a git repository booting a rekcurd service. (GitOps option)
But general usage for Kubernetes is using an image.

Goal

• Add image driven boot option

Implementation

• implement
• testing

Support different Kubernetes cluster but same application

Motivation

TBD

Goal

• TBD

Implementation

• TBD

Handle parallel request return status

Motivation

Dashboard cannot handle multiple Rest API response statuses.
See here.

Ideally, dashboard must popup all API response status or all failure response status.

Goal

• Handle parallel request return status

Implementation

• TBD

Frontend unittest

Motivation

Frontend unittest
https://mae.chab.in/archives/60066

Implementation

• testing

Monitor Kubernetes statuses

Motivation

To monitor the status of Kubernetes cluster/pod/etc.., we need a visualization function.

Goal

• Implement Kubernetes monitoring

Implementation

• TBD

evaluation data versioning

Motivation

support versioning for evaluation data to save volume usage when applying small change.

Goal

• support versioning for evaluation

Implementation

• save whole git data to data server
• able to show diff

Uploading model sometimes returns errors.

Motivation

When Uploading model by requesting directly to API (not from frontend), sometimes return errors.
The model is actually uploaded, but API returns error response

Log in client:

{"message": "(pymysql.err.OperationalError) (2006, \"MySQL server has gone away (ConnectionResetError(104, 'Connection reset by peer'))\") [SQL: 'SELECT services.service_id AS services_service_id, services.service_name AS services_service_name, services.display_name AS services_display_name, services.application_id AS services_application_id, services.model_id AS services_model_id, services.service_level AS services_service_level, services.register_date AS services_register_date, services.confirm_date AS services_confirm_date, services.update_date AS services_update_date, services.host AS services_host, services.description AS services_description \\nFROM services \\nWHERE services.application_id = %(application_id_1)s \\n LIMIT %(param_1)s'] [parameters: {'application_id_1': 10, 'param_1': 1}] (Background on this error at: http://sqlalche.me/e/e3q8)"}
INFO - {models.py:1736} ERROR - 500:INTERNAL SERVER ERROR

Goal

• finish uploading model successfully and the API returns the correct response.

Implementation

Istio 1.1 support

Motivation

https://istio.io/about/notes/1.1/

Goal

• write the goal

write the description

Implementation

• feature
• feature
• resource
• testing

Fixed ssh directory mount path

Motivation

The dashboard will mount the ssh directory to the container for retrieving the git repository. However, the mount path is fixed to /root/.ssh/ but there are some platforms(e.g. GCP) which /root directory is read-only. The failure of mounting ssh volume will cause the deployment of the services to fail.

Goal

Users should be able to configure the mount path by using frontend and API. I'm also wondering if it's possible to use Kubernetes secret to mount.

Implementation

Please leave your comments if I missed something important.

• Add ssh related fields to the Kubernetes object
• Make Kubernetes API be able to create or update a Kubernetes object with specified ssh mount path
• Use ssh related fields to create or update services in create_or_update_drucker_on_kubernetes
• Add input textbox on frontend site
• Testing

Thank you!

Add access control to Kubernetes

Motivation

Need an access control for Kubernetes which dashboard manages.

Add Breadcrumb list

https://reactstrap.github.io/components/breadcrumbs/

allow to add Service without any models in Local mode

Motivation

• In Local mode, cannot add model when no services are deployed
• currently, cannot add service when no models are uploaded

So only in local mode, allow to add Service without any models in Local mode

Goal

• able to add service in Local mode

Implementation

• in local mode, able to display page to add service with no model uploaded
• in local mode, not have to select model when adding service
• in local mode, raise error when uploading model with no service deployed
  • currently, add record to db and return 200 (not save model content anywhere)

Azure support

Motivation

Azure support

Implementation

• AKS support
• Azure storage (BLOB?) support
• testing

Refactoring around authentication frontend

Motivation

Refactoring frontend codes around LDAP authentication.

Goal

• Fix error handling more generally without exceptional if statement
• Implement Authentication component which wraps all other components
  to handle authentication

Add storage options for ML models and evaluation data

Motivation

Current architecture requires that nodes must mount cloud storage. This spec is not developer-friendly.
Adding storage option will support developers to use Rekcurd easier.

Goal

• Get models via APIs.

Implementation

• Use boto to connect ceph s3 APIs
• Use boto3 to connect AWS S3 APIs
• Use google-cloud-storage to connect GCS APIs
• testing