minio / m3

MinIO Kubernetes Cloud

m3 (mkube)

MinIO Kubernetes Cloud

Installation

You can apply all the files located in k8s/base/ to install m3

kubectl apply -f k8s/base/

Or you can use kustomize to build a single file to apply which supports customizations

kustomize build k8s/base/ | kubectl apply -f -

Development

If you want to do some development for m3 please refer to our Development document

m3's Issues

Profiling start and stop for mcs

Im implementing the start and stop profiling endpoint for mcs based in the following swagger definition

  /api/v1/profiling/start:
    post:
      summary: Start recording profile data
      operationId: ProfilingStart
      parameters:
        - name: body
          in: body
          required: true
          schema:
            $ref: '#/definitions/profilingStartRequest'
      responses:
        201:
          description: A successful response.
        default:
          description: Generic error response.
          schema:
            $ref: "#/definitions/error"
      tags:
        - AdminAPI
  /api/v1/profiling/stop:
    post:
      summary: Stop and download profile data
      operationId: ProfilingStop
      produces:
        - application/octet-stream
      responses:
        201:
          description: A successful response.
          schema:
            type: file
        default:
          description: Generic error response.
          schema:
            $ref: "#/definitions/error"
      tags:
        - AdminAPI

  profilingType:
    type: string
    enum:
      - cpu
      - mem
      - block
      - mutex
      - trace
      - threads
      - goroutines
  profilingStartRequest:
    type: object
    required:
      - type
    properties:
      type:
        $ref: "#/definitions/profilingType"

Node management

We need a set of APIs to define and manage the underlying infrastructure to onsite deployments. This should be accessible via CLI and API

The tables that hold the data looks as follows:

We need support to:

• Add nodes CLI
• Add nodes API
• List Nodes CLI
• List Nodes API
• Get details for a Node CLI
• Get details for a Node API
• Add drives to a node CLI
• Add drives to a node API
• List Drives on a Node CLI
• List Drives on a Node API
• Get details for a drive CLI
• Get details for a drive API

Allocation

We need an algorithm that decides where to place a brand new tenant based on certain allocation rules:

• Storage Group is not full
• Storage Cluster has space and below 85% threshold
• If no storage group is available, create a new SG inside a SC with available space

Add grafana deployment for central dashboard?

Not sure shall we do grafana for central dashboard to scrape all prometheus annotations?

https://grafana.com/grafana/plugins/grafana-kubernetes-app

Generate minio service account credentials for debugging

We need to add an operator command for generating tenant service account credentials for debugging purposes.

Abstract Tenant MinIO Configuration as a secret

We need to move the tenant environment variables, currently stored as a Config Map, to a kubernetes secret

Move Storage to it's own namespace

We need to move all the storage to it's own storage group so we can control access to and from it using kubernetes policy and network policies

Error when try to assign the same 2 permissions to a service account

When updating service account permissions that contains the same bucket rule, ie allow read access to same bucket in two different permissions this error happens:

2019/12/11 01:50:32 MinIO: duplicate actions [s3:GetBucketLocation s3:GetObject s3:ListBucket s3:PutObject], resources [arn:aws:s3:::movies/*] found in statements { Allow [s3:GetBucketLocation s3:GetObject s3:ListBucket s3:PutObject] [arn:aws:s3:::movies/* arn:aws:s3:::music/*] []}, { Allow [s3:GetBucketLocation s3:GetObject s3:ListBucket s3:PutObject] [arn:aws:s3:::movies/*] []}

Integrate Prometheus into Tenant MinIO

We need to integrate Prometheus with the Tenant MinIO to keep track of usage metrics

• Provision a Prometheus along with the Tenant (cmd tenant add)
• Configure the Tenant MinIO to send it's data to Prometheus via the Tenant MinIO Configuration Secret

Timeout when adding a new user to a tenant

When trying to add a new user to a tenant via:

./m3 tenant user add --tenant TENANT --email [email protected] --name somename --password somepassword

We are getting this error after a timeout:

Error adding user: Put http://XXXXXX-sg-1:9001/minio/admin/v1/add-user?accessKey=XXXXXXXX: dial tcp: lookup XXXXXX-sg-1: no such host

However the user is created correctly

Error when trying to run migrations

When there is new migrations to apply to the database, i try to run them with ./m3 setup db

./m3 setup db --help
NAME:
  m3 setup db - runs DB migrations

USAGE:
  m3 setup db [command options] [arguments...]

FLAGS:
  --help, -h  show help

They failed with error:

pq: schema "provisioning" already exists
pq: database "tenants" already exists
Adding the first admin
Error adding user: pq: duplicate key value violates unique constraint "admins_email_uindex"
admin m3 error
pq: duplicate key value violates unique constraint "admins_email_uindex"

Seems like the command is trying to do the whole setup db process instead of just running the missing migrations.

Username is mandatory when adding name to a tenant but there is no name flag

When trying to add a new user to a tenant via cli username is mandatory but there is no flag to specify the name

./m3 tenant user add --tenant company-inc --email [email protected] --password user1234

Error:

User name is needed

disable user public handler should invalid the user session

today enable/disable user public api handler is only modifying users database
it should call:

err = cluster.UpdateSessionStatus(appCtx, sessionRowID, "invalid")
	if err != nil {
		return nil, status.New(codes.Internal, err.Error()).Err()
	}

Introduce support for transaction-like changes to Kubernetes

We need to keep track of the Kubernetes changes we are making along an operation, so if something else fails along the way, we can roll back the k8s operations.

We can use the cluster.Context to keep track of k8s changes we are doing, and how to roll them back.

An example is adding a tenant, we create config maps and then service and lastly deployment, but if anything fails, and the command fails, the first two successful commands stay on k8s therefore blocking other subsequent commands.

Implement local user IDP

We need some basic User management features:

• Register a User - return JWT token for the session
• Login as a User - return JWT token for the session
• Validate User Email for the provided Tenant (company)

Send Emails

• Configure external email provider
• Send emails internal api

Reconnect m3 login page

This ticket needs to focus on 4 things:

• Remove the field for company
• Change email & password fields to be Access & Secret
• Connect the form to the updated API for login
• Store the token in a local storage variable & redirect

Upgrade Storage Group's MinIO

We need an algorithm that updates a Storage Group by stages

1. drain the pod within the SG that is going to be updated by updating each tenant service pointing to this SG
2. update the pod with the new MinIO version
3. wait for the pod to be ready, all containers should pass readiness check
4. re-enable the pod within each tenant service
5. repeat for the next pod

API Validate Tenant Short Name

We need an API to Validate Tenant Short Name, we already have a proper internal function to validate, just expose it via API

Validate Tenant Short Name on creation

We need to make sure the tenants being created have a valid short name as that's used as identifier and subdomain.

Add this validation as a function so it can be re-used in many places.

A valid tenant short name must:

• Is globally unique
• min 2 characters
• max 64 characters
• is only lowecase, numbers and dashes [a-z0-9-]{2,64}
• Not a reserved word: m3, kube-system, default, kube-public, kube-node-lease

Integrate any KMS

• Provision a KMS per tenant
• Configure MinIO to use the KMS
• Store the configuration on the Tenant MinIO Configuration (Secret)

pq syntax error when creating a tenant with -

On a brand new cluster when creating the tenant following the README example we are getting this error:

./m3 tenant add company-name
Registered as tenant 244cb114-1737-438e-ad4c-2777acdde5d8

done creating tenant service for tenant company-name-sg-1
sql: transaction has already been committed or rolled back
pq: syntax error at or near "-"

Manage Storage Clusters

We need a set of APIs to manage the layout of a storage cluster, which nodes participate in the cluster and how many storage groups are inside the storage cluster

Tasks:

• Get a list of Storage Clusters and their details (nodes, drives, storage groups)
• Get SC usage (sg, tenants, usage)
• Lock a Storage Cluster. Prevent new SG or Tenants from being added.
• Get a list of Storage Groups and their details (tenants)
• Create a new Storage Group inside a Storage Cluster

Add securityContext for the pods as default

https://kubernetes.io/docs/tasks/configure-pod-container/security-context/ to allow MinIO to run as non-root on these pods

Implement Main LB router

We need to implement a router that can route to any of the valid tenants on the cluster via it's named services and ports

• Resolve TENANT-SHORT-NAME.app-domain.com to the proper Tenant identified by the shortname
• When a Tenant signs up update the routing rules
• Avoid dropping connections during reconfig

mcs make bucket api returns error if access not set

Error returned 500 if access not specified on make bucket request POST api/v1/buckets
Proposals

• either we make access required on the request and delete the bucket if something failed after creating it or:
• we split this into two separate call this requires change on the wireframes

Node capacity/usage

We need a function that queries the registered storage nodes and their drives and figures out how much space is left.

Copy m3's node selector (if present) to the m3-jobs

Apply changes to Deployments as a merge operation instead of Delete/Create

Right now, when we are re-deploying a Tenant we are deleting the deployment for the Storage Group and then re-creating it (Delete->Create), this has a problem that we loose deployment metadata.

We should instead do Get->Update to preserve other metadata that the deployment may have.

Add informers and existing validation to the setup process

We need to add informers to track the deployment/configuration process to the following items during the ./m3 setup process:

• setup namespace
• setup nginx Load balancer
• setup postgres
• Setup jwt secrets

Context deadline exceeded message is shown when no valid dockerhub image is set for new tenant

When you try to create a new tenant with an incorrect / empty dockerhub image, a timeout error is thrown:

"Get "https://dl.min.io/server/minio/release/linux-amd64/": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"

Also this error is being sent as a 200 response so the application thinks that everything went OK.

We need to set a default image for these cases and fix the error response code to be something different than 200