Unsupported

This project is Unsuppoted by MongoDB. Alternative supported products for managing MongoDB Atlas clusters include:

~~ For Reference Only ~~

Introduction
Getting Started
Specifications
VMWare Tanzu Application Service
Notes
License
Development

This project is a new version of the original MongoDB Atlas Service Broker called "atlas-osb"

Atlas-osb adds the ability to define your own plans from Atlas resource templates. This new and powerful feature brings the broker to a new level of production-readiness. One simple "create service" command is all you need to provision a complete Atlas deployment including a Project, Cluster, Database user, firewall access, and more.

Use the Atlas Service Broker to connect to MongoDB Atlas from any platform which supports the Open Service Broker API, such as Kubernetes and Pivotal Cloud Foundry.

Provision managed MongoDB clusters on Atlas directly from your platform of choice. Includes support for all cluster configuration settings and cloud providers available on Atlas.
Manage and scale clusters without leaving your platform.
Create bindings to allow your applications access to clusters.
Securely configure and deploy the broker with multiple Atlas apikeys integrated with systems such as Pivotal CredHub and Kubernetes Secrets.

atlas-osb custom plans allow cluster administrators to manage real-world production scenarios for the MongoDB Atlas Platform within any OSB-compliant environment. Custom plan's are templates of Atlas API resources.

Custom plans use the Atlas API resources and allow users to define reusable packages of related Atlas resources. These are known as "plan's". Cluster administrators can deploy these plans into Open Service Broker API marketplaces. The plans are then available through standard service broker catalog integration. This is available in Cloud Foundry, Kubernetes via Service Catalog, and other modern containerized computing environments.

Getting Started

Prereq's for Cloud Foundry

Typical cli/bash commands are used for this procedure along with the following tools:

cf Cloud Foundry cli
hammer cli (recommended)
text editor
MongoDB Atlas account - you need an Org apikey

Once you have your workstation ready, head over to http://cloud.mongodb.com and create an organization apikey. For more step by step details on this step, step over here: https://docs.atlas.mongodb.com/configure-api-access/#manage-programmatic-access-to-an-organization.

Installation & Configuration

Pull down the latest release of the atlas-osb (recommended).

curl -L $(curl https://api.github.com/repos/mongodb/atlas-osb/releases/latest | grep tarball_url | awk '{print $2}' | tr -d '",') -o atlas-osb-latest.tar.gz
tar xvf atlas-osb-latest.tar.gz
cd atlas-osb

or, you can clone this repo.

Setup apikeys. This flow uses a simple User Provided Service to store the apikey for the broker. The broker also supports CredHub integration [TODO: add section].

Copy this template and update with your own apikey information.

{
   "broker": {
      "username": "admin",
      "password": "admin",
   },
   "keys": {
      "<KEY-NAME>": {
        "publicKey": "<PUBLIC-KEY>",
        "privateKey": "<PRIVATE-KEY>",
        "orgID": "<ORG-ID>",
      }
   }
}

Save this in a file called keys.

Create a user provided service:

# create simple user-provided-service with keys file
cf cups atlas-osb-keys -p keys

Installing the service broker app

The initial setup includes deploying the cf app and configuring some environment variables. By default, the broker will only listen on localhost (127.0.0.1) and port 4000. Use the set-env command to update this for Cloud Foundry defaults (port 8080).

# push the cf app for the broker
cf push atlas-osb --no-start

# connect the keys to the broker & configure settings
cf bind-service atlas-osb atlas-osb-keys

# use the provided sample plan templates
cf set-env atlas-osb ATLAS_BROKER_TEMPLATEDIR ./samples/plans

cf set-env atlas-osb BROKER_HOST 0.0.0.0
cf set-env atlas-osb BROKER_PORT 8080

# start it up
cf start atlas-osb
```bash

Check the logs, you should see output included the loaded plan templates.

```bash
cf logs atlas-osb --recent

Grab the routes value from cf app atlas-osb and use this URL in the following command.

# create the actual broker
cf create-service-broker atlas-osb admin admin <YOUR-DEPLOYMENT-URL>
# enable access
cf enable-service-access atlas

Check out the results.

cf marketplace
cf apps

Create an Atlas cluster

For this flow, we'll create an instance of the "basic-plan" called "hello-atlas-osb". See /samples/plans/sample_basic.yml.tpl.

cf create-service atlas basic-plan hello-atlas-osb
cf service hello-atlas-osb

Head over to the Dashboard URL returned from the last command (http://cloud.mongodb.com/...) to follow progress as your Atlas resources are provisioning.

(Optional) You can test connecting to your new Atlas cluster with a very simple app in package.

Push, bind, and test sample app.

cd test/hello-atlas-cf
cf push hello-atlas-cf --no-start
cf bind-service hello-atlas-cf hello-atlas-osb
cf start hello-atlas-cf
cf app hello-atlas-cf

Grab the route and load it up in your browser. You should see connection information to your new Atlas cluster.

How to bind and connect to a specific database

You need to override the role when creating the binding like this:

 cf bind-service hello-atlas-cf wed-demo-1 -c '{"User" : {"roles" : [ { "roleName" : "readWrite", "databaseName" : "mydatabase"} ] } }'

You can check this works with the hello-atlas-cf app with something like this:

cf app hello-atlas-cf | grep routes | cut -d: -f2 | xargs -I {} google-chrome "{}"

Or you can call create-service-key command:

cf create-service-key service-instance-name key-name -c '{"user" : {"roles" : [ { "roleName" : "readWrite", "databaseName" : "mydatabase"} ] } }'

Pausing a cluster

 cf update-service <SERVICE-INSTANCE-NAME> -c '{ "paused":true }'

Note - do not put quotes around the true/value.

Updating a cluster

First - note not all possible updates are supported at this time. Some types of updates (i.e. project/cluster renaming) are not supported by Atlas at all.

In general, to update a plan instance:

 cf update-service -b dyno-ss mongodb-atlas-template basic-plan dyno-ss-4oh -c '{ "cluster" :  { "mongoDBMajorVersion": "4.0" } }'

Updating service instance dyno-ss-4oh in org atlas-broker-demo / space dynoss as admin...
OK

The syntax for the -c json you send in create-service or update-service is processed in two ways:

The document passed is parsed and matched into the template dot-variables, then the template is executed.
The passed document treated as a partial plan-instance and then merged into the results from step 1.

This allows service settings to be updated.

Specification

Configuration Reference

Configuration is handled with environment variables. Logs are written to stdout/stderr as appropriate and each line is in a structured JSON format.

Variable	Default	Description
`ATLAS_BASE_URL`	`https://cloud.mongodb.com/api/atlas/v1.0/`	Base URL used for Atlas API connections
`REALM_BASE_URL`	`https://realm.mongodb.com/api/admin/v3.0/`	Base URL used for Realm API connections
`BROKER_HOST`	`127.0.0.1`	Address which the broker server listens on
`BROKER_PORT`	`4000`	Port which the broker server listens on
`BROKER_LOG_LEVEL`	`INFO`	Accepted values: `DEBUG`, `INFO`, `WARN`, `ERROR`
`BROKER_TLS_CERT_FILE`		Path to a certificate file to use for TLS. Leave empty to disable TLS.
`BROKER_TLS_KEY_FILE`		Path to private key file to use for TLS. Leave empty to disable TLS.
`BROKER_APIKEYS`		Path to file or JSON string containing credentials.
`ATLAS_BROKER_TEMPLATEDIR`		Path to folder containing plans e.g. ./samples/plans

The values for the OSB "Service" for a given atlas-osb instance can be customized with a set of additional environment variables. Each of these are optional, and has default content. Here are the settings for the domain.Service returned for each plan:

OSB Service Metadata Configuration Reference

domain.Service:

Field	Env variable	Default
`ID`	NONE	`"aosb-cluster-service-template"`
`Name`	`BROKER_OSB_SERVICE_NAME`	`"atlas"`
`Description`	`BROKER_OSB_SERVICE_DESC`	`"MongoDB Atlas Plan Template Deployments"`
`Metadata.DisplayName`	`BROKER_OSB_SERVICE_DISPLAY_NAME`	`"MongoDB Atlas - %s"` with `"Template Services"`
`Metadata.ImageUrl`	`BROKER_OSB_IMAGE_URL`	`"https://webassets.mongodb.com/_com_assets/cms/vectors-anchor-circle-mydmar539a.svg"`
`DocumentationUrl`	`BROKER_OSB_DOCS_URL`	`"https://support.mongodb.com/welcome"`
`ProviderDisplayName`	`BROKER_OSB_PROVIDER_DISPLAY_NAME`	`"MongoDB"`
`LongDescription`	`BROKER_OSB_LONG_DESC`	`"Complete MongoDB Atlas deployments managed through resource templates. See https://github.com/mongodb/atlas-osb"`
`Metadata.Tags`	`BROKER_OSB_SERVICE_TAGS`	`"mongodb"`

Multiple API Key Support

These are the requirements for supporting multiple apikeys. Note: Sometimes we use the term "CredHub" to refer to this overall feature to support multiple keys.

The broker will accept a json/yaml object containing authentication credentials. The format for this will be:

# broker is a dictionary with the HTTP Basic Auth credentials for requests to service-broker
broker:
  username: 'admin'
  password: 'admin'
# keys is a dictionary with Atlas API Keys in mostly native format
keys:
  first-key:
    desc: 'the first key' # optional, for user reference only
    publicKey: '12345'
    privatekey: '12345'
    orgID: '12345'
  the 2nd key:
    publicKey: '12345'
    privateKey: '12345'
    orgID: '545454'

Where each key conforms to the following schema defined by the ApiKey struct, with a few changes:

an additional orgID field is used as fallback when a plan template has no explicitly selected key
id and desc are optional and are not currently used for anything

Multi-apikey support requirements (P# where # is priority with 0 highest.)

P0 Reading keys as a string from the BROKER_APIKEYS environment variable.
P0 Reading path from the BROKER_APIKEYS environment variable and then loading from file.
P0 Reading from Cloud Foundry VCAP_SERVICES to support CredHub integration.
P1 Support reading from Kubernetes Secrets.
The broker will allow selection of appropriate apikey during plan provisioning by discovery of the ApiKey in the plan-template instance context (see below Plans).

Atlas Plan Templates

Providing support for all the myriad of combinations of Atlas resources and arbitrary validation logic is not possible with the current broker "service"/”plan” design. atlas-osb introduces an additional way to define new “services” and “plans” which represent arbitrary Atlas resource objects (object graphs) by leveraging the declarative design of the Atlas API and YAML templates.

Most users and most apps need the following resources at minimum for typical usage:

1 Atlas Project to contain everything
1 Standard 3-node replica set for your database
1 IP-Whitelist so that your app can connect
1 DB credential, again, so your app can connect

We can model the above with the following set of templates:

Reference Sample Basic Full Example

name: basic-plan
description: This is the `Basic Plan` template for 1 project, 1 cluster, 1 dbuser, and 1 secure connection.
free: true
apiKey: {{ keyByAlias .credentials "testKey" }}
project:
  name: {{ .instance_name }}
  desc: Created from a template
cluster:
  name: {{ .instance_name }}
  providerBackupEnabled: {{ default "true" .backups }}
  providerSettings:
    providerName: {{ default "AWS" .provider }}
    instanceSizeName: {{ default "M10" .instance_size }}
    regionName: {{ default "US_EAST_1" .region }}
  labels:
    - key: Infrastructure Tool
      value: MongoDB Atlas Service Broker
databaseUsers:
- username: {{ default "test-user" .username }}
  password: {{ default "test-password" .password }}
  databaseName: {{ default "admin" .auth_db }}
  roles:
  - roleName: {{ default "readWrite" .role }}
    databaseName: {{ default "default" .role_db }}
ipAccessLists:
- ipAddress: "0.0.0.0/1"
  comment: "everything"
- ipAddress: "128.0.0.0/1"
  comment: "everything"

Requirements

P0 Support loading Plan templates from json or yaml files mounted into the broker runtime at deployment-time.
The broker should support reading plan templates directly from Linux environment variables.
1. P0 Support a default directory to load templates from ATLAS_BROKER_TEMPLATEDIR
ATLAS_BROKER_TEMPLATEDIR = "/templates" UPS - user provided service to mount files into /templates K8s - mount configmaps as files, etc..., Docker, etc..
Include loaded templates with INFO level and also full json payload logging to broker logs (to help debugging).
Always reload templates on startup (restart to deploy new templates)
Template should be standard go-templates and support typical values/variable style replacements.
Users should be able to specify template parameters during service provisioning or update.
Provide a set of common resource yaml/json samples and templates.
Allow reading apikeys and group/org ids from the "CredHub" multi-apikey support
1. Make a template variable called credentials available which contains all the apikeys
Allow reading apikey from a yaml/json file with a resource definition.
🚧 Support a --dry-run flag whenever processing a provision, update, or delete operation on a custom plan. Default is false. Include pre & post template processing in logging output. (Not supported yet.)

Plan Functional Design

Each Plan instance is managed through the OSB provision, bind, unbind, and deprovision operations. In this section we describe the relationship between a Plan's OSB operations and how the broker translates them to Atlas Client API calls.

🚧

Provisioning & Deprovisioning

When the broker gets a call to provision a plan, it will iterate through the various Atlas resources in the plan can call the corresponding service Create method. Similarly, when deprovisioning, the broker will delegate calls to the Atlas Go-client corresponding service Delete function.

Plans are loaded at startup and first validated before being made available in the Marketplace.

read plans from disk
do a first-pass template parse with empty context (substitute "" for everything in template)
- default values can be supplied using {{ default "default-value" .dynamic_value }}
- 🚧 Provide a way to test the plan actually works (dry-run)
parse the result into yaml, get static metadata like plan name, description, instance size, whatever
on provision, do a parse with full context - this is the final plan spec
- 🚧 Allow for dry-run at this step too.

Managing State

This section describes how the state of plan definitions and service instance metadata will be stored.

atlas-osb introduces the use of MongoDB Realm applications to manage the state of your deployed service instances. This happens automatically, directly through the Atlas and Realm APIs.

The atlas-osb will create special project called "Atlas Service Broker Maintenance" for each Atlas Organization. Here, the broker will create a Realm App called "Broker State" 🚧 and use Realm Values to store reference data for the service instances you have deployed across your environments. This approach has several advantages:

zero-footprint on OSB-marketplace environment
single location for all service instance metadata
follows the Atlas organization boundaries, allows support for multi-tenancy scenarios
automatically encrypted via Realm

See Realm Values & Secrets

Bind & Unbind

The OSB bind function is used to provision a new database user credential and connection information for an application using MongoDB. This usually happens when an app is deployed into a new environment. To support this, the broker will create new Atlas resources for the binding and return the connection information appropriately.

🚧 broker needs refactor to use Plan user/ipwhitelist.

When the broker creates a binding, it will translate the Connection Details for the given cluster into the OSB Binding structure.

Connection Details types.

The format for the JSON available for binding in VCAP_SERVICES is:

{
  "connectionString": "mongodb+srv://uuuuuuuuu:[email protected]/admin",
  "password": "xxxxxxxxxx",
  "uri": "mongodb+srv://chewy-123.6bikq.mongodb.net",
  "username": "uuuuuuuuuu",
  "database": "admin"
}

Please see the test/hello-atlas-cf sample app to see details on the binding information available to apps.

Overriding the database for all bindings

Certain customers may wish to control the exact name of the database to which apps using Atlas services can use. This is controlled by inserting the database name into the connection string (as the last forward-slash piece before the query string) which is constructed during a call to the brokers Bind function.

In general, we do not recommend using this feature. However it will be released as "Deprecated" in the Beta release for the broker. Customers are encouraged to refactor their dependencies on fixing this database name.

Default Bind Roles

For example, to set the default role for any database users created via the bind() OSB operation, add these to the settings for your plan.

name: basic-overrides-plan
description: This is an extension of the `Basic Plan` template for 1 project, 1 cluster, 1 dbuser, and 1 secure connection. But it added the ability to override the bind db.
free: true
apiKey: {{ keyByAlias .credentials "testKey" }}
settings:
  overrideBindDB: "products"
  overrideBindDBRole: "readWrite"
project:
  name: {{ .instance_name }}
  desc: Created from a template
...

This sets gives readWrite on the products database for each new user created via bind().

Plans and Atlas Resource Types

The following types are supported for loading from multiple or a single yaml or json objects.

Note these types are derived from the following specifications:

Each "plan" is loaded from a single json/yaml object describing the resources managed by the plan. Plans can contain any number of various MongoDB Atlas resources, but require an Atlas Project. Since each Atlas resource is always associated with an Atlas Project, each Plan must have a project. The other resources are optional.

The Plan and Binding types are new structures to implement this new feature, while the other types are used directly from the MongoDB Atlas Go client.

The officially supported Atlas Resources are:

Project
Cluster
DatabaseUser
ProjectIPAccessList

Plan

This represents an Open Service Broker plan (NOTE: https://github.com/openservicebrokerapi/servicebroker/blob/master/spec.md#service-offering-object)

Please see the basic sample template for a documented ready-to-use template.

// Plan represents a set of MongoDB Atlas resources
type Plan struct {
	Version       string                              `json:"version,omitempty"`
	Name          string                              `json:"name,omitempty"`
	Description   string                              `json:"description,omitempty"`
	Free          *bool                               `json:"free,omitempty"`
	APIKey        *credentials.APIKey                 `json:"apiKey,omitempty"`
	Project       *mongodbatlas.Project               `json:"project,omitempty"`
	Cluster       *mongodbatlas.Cluster               `json:"cluster,omitempty"`
	DatabaseUsers []*mongodbatlas.DatabaseUser        `json:"databaseUsers,omitempty"`
	IPAccessLists []*mongodbatlas.ProjectIPAccessList `json:"ipAccessLists,omitempty"`
	Settings      map[string]interface{}              `json:"settings,omitempty"`
}

The remaining resource type definitions are taken directly from the Atlas Go Client, and therefore subject to change per that project.

Project

An Atlas project (NOTE: https://docs.atlas.mongodb.com/reference/api/projects/). Requires an organization id or valid context from apikey.

Project type.

Cluster

An Atlas cluster (NOTE: https://docs.atlas.mongodb.com/reference/api/clusters-create-one/#example-request).

Cluster

TODO: MARK WHICH FIELDS ARE READ-ONLY? ie. users need to understand what can be in the template

Database User

Database_Users

Project IP Access List

Project_IP_Access_List

VMWare Tanzu Application Service

Test Status

Product Snapshot

The following badges provide version and version-support information about Atlas-OSB for VMware Tanzu.

Notes

Deploying Atlas-OSB to CF. There are several ways to deploy atlas-osb to cloud foundry:

with docker image

cf push APP_NAME --docker-image [REGISTRY_HOST:PORT/]IMAGE[:TAG] [--docker-username USERNAME] [-c COMMAND] [-f MANIFEST_PATH | --no-manifest] [--no-start] [--no-wait] [-i NUM_INSTANCES]

with manifest (to specify a manifest use -f with the path to a manifest)

Please note that Atlas-OSB does not support the free tier of cluster creation: M0 (Atlas API doesn't have such support)

License

See LICENSE. Licenses for all third-party dependencies are included in notices.

classicvalues / atlas-osb Goto Github PK

atlas-osb's Introduction