openebs-archive / maya

Manage Container Attached Storage (CAS) - Data Engines in Kubernetes

Overview

OpenEBS control plane components like provisioners and operators were hosted in this repository.

As the OpenEBS community started to add new engines, the engine specific control plane components have been moved to their respective repositories.

• cStor operators and CSI driver have moved to openebs/cstor-operators and openebs/cstor-csi respectively.
• Jiva operator and CSI driver have moved to openebs/jiva-operator
• Local PV Hostpath and Device provisioner has been moved to openebs/dynamic-localpv-provisioner
• Jiva and cStor prometheus metrics exporter been moved to openebs/m-exporter
• mayactl for displaying status of Jiva and cStor volumes is merged into openebs/openebsctl

This repository mainly contains code required for running the legacy cStor and Jiva pools and volumes like:

• m-apiserver - used for provisoining the legacy cStor and Jiva pools and volumes.
  • mayactl - packaged along with m-apiserver for fetching the legacy cStor and Jiva volume status.
• admission-server - used for validating Jiva and cStor pool and volume requests.
• m-upgrade - used for upgrading the legacy Jiva volumes, cStor pools and volumes.
• cstor-pool-mgmt and cstor-volume-mgmt - used for managing the legacy cStor pool and volumes.

With OpenEBS 3.0, all of the above legacy components are deprecated and users are requested to migrate towards using:

• CStor CSI Driver
• Jiva CSI Driver

The steps to migrate are provided here: https://github.com/openebs/upgrade.

v2.12.x is the last active branch on this repository, that will be used to mainly resolve any security vulnerability or kubernetes compatibility issues found on production setups using the legacy provisioners. New features will be developed only cStor and Jiva CSI drivers.

Install

Please refer to our documentation at OpenEBS Documentation.

Release

Prior to creating a release tag on this repository on v2.12.x branch with the required fixes, ensure that:

• the dependent data engine repositories and provisioner are tagged.
• update the versionDetails.go to include the supported upgrade path.

Once the code is merged, use the following sequence to release a new version for the legacy components:

• New release tag on v2.12.x branch of openebs/linux-utils
• (If required) New release tag on v0.6.x branch of openebs/ndm
• New release tag on v2.12.x branch of openebs/cstor and openebs/libcstor
• New release tag on v2.12.x branch of openebs/jiva
• New release tag on v2.12.x branch of openebs/openebs-k8s-provisioner
• New release tag on v2.12.x branch of openebs/m-exporter
• New release tag on v2.12.x branch of openebs/maya
• New release tag on v2.12.x branch of openebs/velero-plugin

Note: The github release workflows are setup to push the tag to the dependent repositories. In the above case, if a release tag is created on v2.12.x branch of linux-utils, then it will trigger the releases down to velero-plugin repo.

Once the tags are generated update the helm charts and YAMLs at:

• https://github.com/openebs/charts/tree/2.x/charts/openebs/templates/legacy
• https://github.com/openebs/charts/tree/main/charts/openebs/templates/legacy
• https://github.com/openebs/charts/blob/gh-pages/legacy-openebs-operator.yaml

Contributing

We are looking at further refactoring this repository by moving the common packages from this repository into a new common repository. If you are interested in helping with the refactoring efforts, please reach out to the OpenEBS Community.

For details on setting up the development environment and fixing the code, head over to the CONTRIBUTING.md.

Community

OpenEBS welcomes your feedback and contributions in any form possible.

• Join OpenEBS community on Kubernetes Slack
  • Already signed up? Head to our discussions at:
    • #openebs
    • #openebs-dev

License

maya's Issues

Make use of golang design linting

This is very similar to #13.

This issue should target linting golang code

Ref - https://blog.gopheracademy.com/advent-2016/some-tools-for-go-that-you-might-not-know-yet/

Install the maya server dependencies via maya cli -- maya install

maya server depends on consul and nomad server being installed. User should be able to trigger the installation of the maya dependencies using "maya install".

• download the required version of the nomad/consul
• configure nomad/consul as primary server or attach to the existing cluster

Tracking installations at various customer deployments

https://github.com/hashicorp/go-checkpoint

Support for EBS compatibile API

The volume creation API should be compatible with EBS.

Support for installing OpenEBS K8s-FlexVolume Driver

User should be able to use maya to install the right version of the FlexVolume Driver on the kubernetes minion hosts, depending on the version of the kubernetes.

The command could look like:
maya install-driver --type=k8s --name=openebs-iscsi

Allow changing of VSM frontend IP address

The iSCSI FrontEnd requires the IP address to be allocated and specified during creation of VSM. There could be a case where the network subnet for storage can change in the environment. Maya should provide a mechanism to change the IP addresses allocated to VSMs.

Introduce CLI to Stop a running VSM

Stop an existing VSM. This command is used to signal allocations to shut down for the given vsm-ID.

maya - provide an option to create the certificates

maya should have option to generate certificates that can be used by a set of hosts (hostname and ip address) forming a kubernetes cluster. These certificates can be used by the k8s master and minion nodes for various services.

certificates and certificate authority files can be created using cfssl, the install and certificate generation steps are as follows:

On any linux client install cfssl

wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64
chmod +x cfssl_linux-amd64
sudo mv cfssl_linux-amd64 /usr/local/bin/cfssl
wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64
chmod +x cfssljson_linux-amd64
sudo mv cfssljson_linux-amd64 /usr/local/bin/cfssljson

Create the files ca-config.json, ca-csr.json and client-csr.json files using the example provided below.

The commands to generate key are :

cfssl gencert -initca ca-csr.json | cfssljson -bare ca
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=client client-csr.json |  cfssljson -bare client

The above steps will generate : ca.pem, ca-key.pem, client.pem, client-key.pem
Rename them as follows to be used during configuration of etcd or k8s services.

mv ca.pem ca.crt
mv ca-key.pem ca.key
mv client.pem client.crt
mv client-key.pem client.key

For example, if the following hosts are used for creating kuberntes master and minion nodes:
k8s-master-1 : 172.28.128.1
k8s-node-1 : 172.28.128.1

The client-csr.json would be like:

{
  "CN": "kubernetes",
  "hosts": [
    "k8s-master-1",
    "k8s-node-1",
    "172.28.128.1",
    "172.28.128.2",
    "127.0.0.1"
  ],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "IN",
      "L": "BLR",
      "O": "Kubernetes",
      "OU": "Cluster",
      "ST": "Karnataka"
    }
  ]
}

The ca-csr.json can be:

{
  "CN": "Kubernetes",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "IN",
      "L": "BLR",
      "O": "Kubernetes",
      "OU": "CA",
      "ST": "Karnataka"
    }
  ]
}

And ca-config.json :

{
  "signing": {
    "default": {
      "expiry": "8760h"
    },
    "profiles": {
      "client": {
        "usages": ["signing", "key encipherment", "server auth", "client auth"],
        "expiry": "8760h"
      }
    }
  }
}

Setting up maya host using setup-osh command fails.

Steps to reproduce:

• Run the command

 maya setup-osh -self-ip=<self-ip> -omm-ips=<master-ip>

Output:

ls: cannot access '/etc/maya.d/scripts/': No such file or directory
Error executing cmd: exit status 2
Install failed: Bootstrap failed: Missing path: /etc/maya.d/scripts/
Get:1 http://security.ubuntu.com/ubuntu xenial-security InRelease [102 kB]
Hit:2 http://archive.ubuntu.com/ubuntu xenial InRelease
Get:3 http://archive.ubuntu.com/ubuntu xenial-updates InRelease [102 kB]
Get:4 http://archive.ubuntu.com/ubuntu xenial-backports InRelease [102 kB]
Fetched 306 kB in 1s (177 kB/s)
Reading package lists...
Reading package lists...
Building dependency tree...
Reading state information...
unzip is already the newest version (6.0-20ubuntu1).
0 upgraded, 0 newly installed, 0 to remove and 31 not upgraded.
Cleaning old maya boostrapping if any ...
Fetching utility scripts ...
Fetching docker scripts ...
Fetching Mayaserver scripts ...
Fetching consul scripts ...
Fetching nomad scripts ...
Fetching Flannel scripts ...
Fetching mayaserver config templates ...
Fetching consul config templates ...
Fetching nomad config templates ...
Error executing cmd: exit status 1
Install failed: Error installing docker
OpenEBS Host setup failed

Test and document the usage of maya on centos

maya cli has been tested on ubuntu, test/investigate if the default release binary works on centos. If the binary doesn't work, check if it can be built from source.

Workflow for launching controller and replica

• Update the launch arguments for replica containers
  • Pass a backing store (default is local directory)
• After a replica is launched, indicate to controller about the replica addition

Maya install - Management, Bare Metal & AWS

Current maya cli based install commands are implemented in a very crude approach.
They are simple wrapper over shell scripts.
However, maya needs to do the overall deploy management (validate, monitor, upgrade, rollback, etc).
In addition it needs to do the deployment over VMs, Bare Metal, & AWS

This issue will serve as the tracking point for all these requirements.

Some References:

• https://github.com/coreos/coreos-baremetal
• https://github.com/coreos/kube-aws

maya - install the k8s flexvolume plugin

Implement a new command that installs k8s flexvolume plugin on the k8s minion nodes.

say, the cli can be like:

maya k8s volume-plugin-install <plugin-name> <path-to-the-volume-plugin-file>

This should install the volume-plugin-file:

cp <path-to-the-volume-plugin-file> /usr/libexec/kubernetes/kubelet-plugins/volume/exec/openebs~<plugin-name>/<plugin-name>
sudo systemctl restart kubelet

Support mayactl option for managing snapshots of a VSM

Setup maya release mechanism

Users should be able to install maya/openebs using the released version, as explained in Step1 of https://github.com/openebs/maya/wiki/Scenario-01---Install

Needs investigation on how release mechanism is enabled via gitub, linking of the released version to a site downloadable file.

Maya install: consider different *nix OS(es) & architecture(s)

vsm-stats is showing zero in output

vsm-stats is showing zero in output, it should be relative.

Maya api server does not provide volume update flow

Things like actual state, desired state, validations, etc should be taken into consideration.

The validations for all CRUD ops in maya api server will make use of 3rd party providers.
e.g. network provider, orchestrators, storage interface agents etc.

Test job specs with replicated jiva container(s) w.r.t their placements

Assumptions:

• Nomad setup in a datacenter with 1 Server & 2 Clients

Steps:

• Create a job specs with 2 tasks
• 1 task will be for jiva
• 2nd task will be for jiva replica
• Each task will reside in different task groups
• Place appropriate constraints that enable placement of these tasks on different nodes

maya - verify the k8s service status

maya version, should be able to detect if the current node is configured as k8s master or minion and print the status of master or minion k8s services.

If the current node is master, print the status of - kube-apiserver, kube-controller-manager and kube-scheduler. On the minion node, print the status of kube-proxy and kubelet.

add tests in vsm-stats

add tests in vsm-stats

Management of persistence store

Refers to issue - #8

Continuous Integration and Deploy Unit Test Cases

Working on testing framework and writing *_test.go files to introduce test cases for better unit testing

Command to setup flannel using maya on OpenEBS Hosts

The following document describes the way to install Flannel on OpenEBS hosts and connect to the etcd that is also managing the k8s - flannel.

https://docs.google.com/document/d/1Ige6SwecYLglxmRT4NDmDr1gYDkoWR_pRF4wiWNA_E8/edit?usp=sharing

The intent is that a command like the below should be support that will automate the steps to install flannel with the required args.

maya setup-nwd flannel <args....>

Duplicate & errorprone export NOMAD_ADDR statements in root's .profile file

The ~/.profile in master-01 VM had both these entries.
The first entry is the right one, whereas the second entry seems to be from some hard-coded stuff.

export NOMAD_ADDR=http://172.28.128.8:4646
export NOMAD_ADDR=http://172.28.128.3:4646

Openebs fails to setup in maya host-02

@AmitKumarDas @prateekpandey14 @kmova ##39

Make vsm-stats interactive

vsm-stats should be interactive. The user has to run maya vsm-stats vol-name every time to see the stats.
You can use curses library for this maybe use https://github.com/nsf/termbox-go

Management of VSM network interface

Refers to issue - #8

Support for maya mount/unmount commands

Add support for the following:

maya mount --type=iscsi --portal=< portal address > --iqn= < iqn > --mntpath=
maya unmount --type=iscsi --mntpath=

These commands can be used from the openebs drivers or just from the command line.

Create a Nomad demo setup - Type 1

Setup Details

• A single nomad server
• 2 nomad clients
• All belonging to a single datacenter as per nomad's nomenclature

support for install/setup of etcd master node

maya cli should include an option to setup etcd-master. The command could be:

maya setup-etcd-master [args....]

arguments could include the listen/expose/advertise ip addresses, peer etcd-master addresses, certificates etc., Use the following steps as a sample for installing the etcd.

sudo apt-get install etcd

Config parameters could be:

ETCD_NAME=k8s-master-1
ETCD_DATA_DIR="/var/lib/etcd"
ETCD_LISTEN_PEER_URLS="https://172.28.128.10:2380"
ETCD_LISTEN_CLIENT_URLS="https://172.28.128.10:2379,http://localhost:2379"
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://172.28.128.10:2380"
ETCD_INITIAL_CLUSTER="k8s-master-1=https://172.28.128.10:2380"
ETCD_INITIAL_CLUSTER_STATE="new"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster-0"
ETCD_ADVERTISE_CLIENT_URLS="https://172.28.128.10:2379"
ETCD_CERT_FILE="/etc/etcd/client.crt"
ETCD_KEY_FILE="/etc/etcd/client.key"
ETCD_TRUSTED_CA_FILE="/etc/etcd/ca.crt"
ETCD_PEER_CERT_FILE="/etc/etcd/client.crt"
ETCD_PEER_KEY_FILE="/etc/etcd/client.key"
ETCD_PEER_TRUSTED_CA_FILE="/etc/etcd/ca.crt"

Setup Maya to Install and configure latest version of Mayaserver

Display the status of the VSM

The status should include the details like:

• IP address
• Node on which the VSM volume is exposed from
• Replica's and their current status (sync/stopped/lagging)

Support for maya vsm-update

NOTE - This will require thinking w.r.t workflow & handling of specs.

Possible workflow:

• Step 1

maya profile-list [prefix]

# Will list various profile available
# Profile are same as specs
# They will be made available during install time
# They may reside at /etc/maya.d/profiles/
# CLI will not have create, update profiles

• Step 2

maya vsm-create -name [options] [path-to-specs]

# path-to-specs can be profile or a custom specs
# path-to-specs is optional, i.e. maya will use the default profile
# name will be override the name provided in the specs at runtime
# *in near future* specs will be stored in git

• Step 3

maya vsm-list [prefix]

# This will list down the vsms that were created earlier

• Step 4

maya vsm-specs -name

# A future requirement
# This will dump the latest vsm specs from git
# This is required for editing the vsm specs before presenting it to maya for an update
# -name signifies that this vsm exists in git (& hence assumed to exist in server)

• Step 5

maya vsm-update -name [options] [path-to-specs]

# -name will be used to search for an existing vsm in git
# path-to-specs will be used to override the existing vsm specs
# path-to-specs can be optional; maya will use the existing specs from git
# diff of desired vs. actual vsm specs (via git) will be displayed
# a dry run of update will be displayed (via server)
# a prompt will be displayed for user confirmation

Update the GO_VERSION in install_go.sh

GO_VERSION should be updated to new stable version 1.8

Allow for running maya master and storage host on a single machine

This is not a recommended configuration, but for development mode and quick prototyping, allow deploying OpenEBS maya master and storage host on a single machine/VM.

Provision the networking scheme for the VSM

Since the replica's and controllers may need to communicate beyond the boundaries of a docker cluster, we will be using "host" based networking scheme. Before a VSM is scheduled, it should be allocated with an IP address for exposing the iSCSI volumes and a set of replica IP/Address and ports.

maya vsm-stop doesn't stop or destroy controller and replica

Issue seen while testing:
https://github.com/openebs/openebs/tree/master/k8s-demo
Doc followed for setup:
https://github.com/openebs/openebs/blob/master/k8s-demo/run-mysql-openebs.md
Commands run inside omm-01 VM:
maya vsm-list
Output:

ID         Type     Priority  Status
demo-vsm1  service  50        running

maya vsm-stop demo-vsm1
Output:

==> Monitoring evaluation "4e35aca5"
    Evaluation triggered by job "demo-vsm1"
    Evaluation status changed: "pending" -> "complete"
==> Evaluation "4e35aca5" finished with status "complete"

maya vsm-list
Output:
No running jobs
But the controller and replica are still up and running inside osh-01 VM.

maya - check the configured cni with docker

on the osh nodes or minion nodes, where the docker is installed. check what docker network plugins (cni) are configured. This can be added as a new command:

maya network status

maya - option to check if a later version is available.

maya version command should also include a note to check if there is a later version available then the one that is currently installed. This will involve contacting the github releases page to get the latest version.

Create a CLI for maya

The primary objectives of the CLI will be:

• It should be intuitive to use
• It should be simple to use
• It should have less learning curve
• It will eventually be inclined to infrastructure storage domain
• It may make use of (i.e. talk to) Nomad CLI or Nomad agent server
• It should not change the way Nomad behaves
• It should compliment Nomad

Install openebs node dependencies via maya cli

Install the requirement components to convert a node into openebs node by installing:

• docker
• nomad/consul client
• configuration of persistent store
• configuration of vsm network interfaces

The location of the persistent store and vsm-network interfaces are specific a node. (For example a one node can have mellanox drivers and other in the same cluster can have intel nics. Though not an ideal case, but stuff happens, when putting together the available hardware into a common cluster.)

The details should be stored in the consul ( DB ) on a per node basis.

maya uses host-based networking for exposing storage. The should be an cli option to specify the network(s) that can be assigned for the VSMs. Whether a network can be used for public (accessing the storage) or private (used for replication). The scope of the networking can be vast, but as part of this issue, it would suffice if the user is able to set the network interface of the node and the range to be used.

New maya cli - Require a command to drain the node from maya cluster

Currently we have a command to add the node to the cluster

$ maya setup-osh -omm-ips -self-ip

There needs to be a command to drain the node from the cluster

$ maya drain-osh <host-id>

error while booting vm

vt-x hardware acceleration disabled and unable to boot vm

Set up Travis CI for maya

maya osh-status gets error at the querying node ..

error querying node status

Use systemd best practices for maya install/deployment related features

Reference -
https://www.digitalocean.com/community/tutorials/how-to-use-systemctl-to-manage-systemd-services-and-units

Maya over Amazon ECS

https://blox.github.io/

maya - check for latest version of nomad/consul

On osh and omm, nomad is installed. On running maya version, verify if there is an update available to the installed nomad and consul versions.

Run shell script linting into existing make target(s)

There are various shell files used in Maya.

• Create a make target to install shell based linting tool
  • i.e. Install https://github.com/koalaman/shellcheck
• Modify make test that will run linting operations on all shell files