stelligent / mu

A full-stack DevOps on AWS framework

Deprecation Notice

Mu is no longer being maintained. We appreciate the support and interest over the past few years. This repository will be archived in the future. Releases will remain available for the foreseeable future. If you are starting a new project on AWS and looking for alternatives to Mu...

We recently did a podcast on Mutato, Mu, and OSS at Stelligent that may be of interest.

Non-Stelligent OSS similar to Mu:

• ECS CLI v2
• CDK - particulary APP Delivery
• AWS SAM
• Service Frameworks
• CDK8s

Additionally, check out RFC 049 of CDK. CDK is where a lot of us are banking the future of AWS deployments are.

These alternatives either didn't exist or were in active development when we started Mu. The increased competition, new ideas, and updates to the ecosystem have made the decision easier to discontinue support for Mu.

Why?

Both Amazon ECS (Elastic Container Service) and Amazon EKS (Elastic Container Service for Kubernetes) provide excellent platforms for deploying microservices as containers. The challenge however is that there is a significant learning curve for microservice developers to deploy their applications in an efficient manner. Specifically, they must learn to use CloudFormation to orchestrate the management of EKS, ECS, ECR, EC2, ELB, VPC, and IAM resources. Additionally, tools like CodeBuild and CodePipeline must be mastered to create a continuous delivery pipeline for their microservices.

To address these challenges, this tool was created to simplify the declaration and administration of the AWS resources necessary to support microservices. Similar to how the Serverless Framework improved the developer experience of Lambda and API Gateway, this tool makes it easier for developers to use EKS or ECS as a microservices platform.

The mu tool uses CloudFormation stacks to manage all resources it creates. Additionally, mu will not create any databases or other AWS resources to support itself. It will only create resources (via CloudFormation) necessary to run your microservices. This means at any point you can stop using mu and continue to manage the AWS resources that it created via AWS tools such as the CLI or the console.

Demo

Watch the 90 second demo below to see mu in action!

Get Started!

Requires AWS Credentials are configured locally (or you are using something like aws-vault).

Install latest version to /usr/local/bin (or for additional options, see wiki):

curl -s https://getmu.io/install.sh | sudo sh

Verify installation

mu -v
mu version 1.5.10

Assuming your project already has a Dockerfile, you can initialize your mu.yml file with: mu init. More details available in the quickstart.

What's next?

Check out the examples to see snippets of mu.yml configuration files that you can use in your own project:

• Basic - Simple website with continuous delivery pipeline deploying to dev and prod environments
• EKS - Demonstration of using EKS provider for mu
• Test Automation - Automating end-to-end testing via Newman
• RDS Database - Defining a database for a service
• Env Variables - Defining environment variables for the service
• HTTPS - Enable HTTPS on the ALB for an environment
• DNS - Associate Route53 resource record with ALB for an environment
• VPC Target - Targeting an existing VPC for an environment
• VPN Connection - Demonstration of adding VPN via CloudFormation
• Custom CloudFormation - Demonstration of adding custom AWS resources via CloudFormation
• Traditional Infrastructure - Demonstration of using EC2 + CodeDeploy rather than ECS for running services

Refer to the wiki for complete details on the configuration of mu.yml and the cli usage:

• Environments - managing VPCs, ECS clusters, container instances and ALBs
• Services - managing ECS service configuration
• Databases - managing database configuration
• Pipelines - managing continuous delivery pipelines
• IAM - managing IAM roles that mu uses
• EKS - using EKS instead of ECS for environment provider
• CLI - details about using the CLI
• Custom CloudFormation - details about customizing the CloudFormation that is generated by mu.
• Service Discovery - details about configuring and using service discovery
• Traditional Infrastructure - details about using traditional infrastructure (EC2 instances) for running services, rather than ECS and Docker.

Support

Need help? Check out the FAQ to try to find an answer to your question. If you can't find an answer there, ask on Gitter!

Contributing

Want to contribute to Mu? Awesome! Check out the contributing guidelines to get involved.

Building from source

• Ensure AWS CLI is configured with an access key, secret access key, and region.
• Install Go tools 1.10+ - (https://golang.org/doc/install)
• If you will be publishing a new release to GitHub, you must install the goreleaser binary for your workstation's OS.
• Clone this repo git clone [email protected]:stelligent/mu.git $GOPATH/src/github.com/stelligent/mu
• Go to src cd $GOPATH/src/github.com/stelligent/mu
• Build with make
• Run unit tests with make test
• Run end-to-end tests with make e2e...takes about 30 minutes and will incur charges in your AWS account.

mu's Issues

List environments

As a developer, I want to be able to list the environments in my account/region so that I can know which environments are currently defined.

AC:

• Running mu env ls will display the following for each environment: environment name, CFN stack name, status, last update, and the version of mu used to create/update stack

Define extension points for plugins

Terminate environment

As a developer, I want to be able to terminate an environment with mu so that I can discard unused resources.

AC:

• Running mu env term <env_name> will teardown the cluster stack
• Will also teardown the VPC stack, unless vpcTarget was specified in mu.yml

APIs for list services

Add service status to show environment

As a developer, I want to be able to view the list of services running in an environment to understand the current configuration and utilization of an environment.

AC:

• When running mu env show <env_name>, mu will list the services deployed to the cluster along with the following: docker image repo, status, last update, mu version

Add extension point for environments

Service build has redundant logs from docker

When running mu svc build you'll see logs repeating from the docker daemon

Add extension point for services

Set service environment variable

Design for CD Pipelines

Would we need to prescribe the standard pipeline or could we support their multi-variant stage pipelines?

How opinionated do we anticipate this to be?

Would they want or would we want to support different types of build, orchestration, and other tools?

Do we need to know anything about their source provider?

Execute tests from CodePipeline

Run CodeBuild to execute tests in test stage of pipeline

Show environment

As a developer, I want to be able to show the details of an environment so that I can understand it's current configuration and utilization.

AC:

• Running mu env show <env_name> will display the stack names for the Cluster and VPC stacks
• Will show base URL for the ALB
• Will list the container instances with the following information: instance id, ami, instance type, AZ, status, # tasks, mem and cpu avail

Vendoring of dependencies

As a mu developer, i want to be able to pin my dependencies to specific version so that I can have deterministic and repeatable builds.

AC:

• should be able to define dependencies with version numbers, ranges
• ci of mu should use dependency definition

Consider: https://github.com/Masterminds/glide

Define the CLI commands

What are all the commands we expect this tool to offer?

Add pipeline status to Show service

As a developer, I want to be able to view the pipeline status for a service to understand the current configuration and utilization of a pipeline.

AC:

• When running mu svc show, mu will list the pipeline stage status for the service along with the following: stage, action, revision, status, last update

Create initial CI job

• Create makefile
• Create Travis or circleci config
• Create unit tests

Blog post on using spring boot plugins

Documentation for installing/using mu

List pipelines

As a developer, I want to be able to list the pipelines in my account/region so that I can know which pipelines are currently defined.

AC:

• Running mu pipeline ls will display the following for each environment: service name, CFN stack name, status, last update, and the version of mu used to create/update stack

Create CONTRIBUTING.md file

Document how to contribute to the project

Undeploy service

As a developer, I want to be able to undeploy a service from an environment so that I can remove services that aren't needed anymore

AC:

• When running mu service undeploy <env_name> the service will be removed from the provided environment
• ECR repo will remain unchanged

APIs for list pipelines

Parse mu.yml file into domain objects

As a mu user, I want to be able to define my service and environments for mu in a YAML file so that I can have a simple definition of my application infrastructure in a file that I can commit with my app to source code.

AC:

• mu.yml would be loaded from current directory by default
• path to mu.yml can be specified via -c flag
• mu should return error if cannot find mu.yml file
• mu should return error if cannot parse yaml in file
• mu should allow the following config options:

---
### Region to utilize
region: us-west-2

### Define a list of environments
environments:

  # The unique name of the environment  (required)
  - name: dev

    ### Attributes for the ECS container instances
    cluster:
      imageId: ami-xxxxxx           # The AMI to use for the ECS container instances (default: latest ECS optimized AMI)
      instanceTenancy: default      # Whether to use default or dedicated tenancy (default: default)
      desiredCapacity: 1            # Desired number of ECS container instances (default 1)
      maxSize: 2                    # Max size to scale the ECS ASG to (default: 2)
      keyName: my-keypair           # name of EC2 keypair to associate with ECS container instances (default: none)
      sshAllow: 0.0.0.0/0           # CIDR block to allow SSH access from (default: 0.0.0.0/0)
      scaleOutThreshold: 80         # Threshold for % memory utilization to scale out ECS container instances (default: 80)
      scaleInThreshold: 30          # Threshold for % memory utilization to scale in ECS container instances (default: 30)

    ### attributes for the VPC to target.  If not defined, a VPC will be created. (default: none)
    vpcTarget:
        vpcId: vpc-xxxxx            # The id of the VPC to launch ECS container instances into
        publicSubnetIds:            # The list of subnets to use for ECS container instances
          - sg-xxxxx
          - sg-xxxxy
          - sg-xxxxz

### Define the service for this repo
service:
  name: my-service                   # The unique name of the service (default: the name of the directory that mu.yml was in)
  desiredCount: 4                    # The desired number of tasks to run for the service (default: 2)
  dockerfile: ./Dockerfile           # The relative path to the Dockerfile to build images (default: ./Dockerfile)
  imageRepository: tutum/hello-world # The repository to push images to and deploy services from.  Leave unset to have mu manage an ECR repository (default: none)
  port: 80                           # The port to expose from the container (default: 8080)
  healthEndpoint: /health            # The endpoint inside the container to determine if the task is healthy (default: /health)
  cpu: 20                            # The number of CPU units to allocate to each task (default: 10)
  memory: 400                        # The amount of memory in MiB to allocate to each task (default: 300)

  # The paths to match on in the ALB and route to this service.  Leave blank to not create an ALB target group for this service (default: none)
  pathPatterns:
    - /bananas
    - /apples

Terminate pipeline

As a developer, I want to be able to terminate a pipeline with mu so that I can discard unused resources.

AC:

• Running mu pipeline term will teardown the pipeline

Cleanup IAM policies

Review the IAM policies and lock down. For sure the service roles for CodeBuild and CodePipeline are too loose at the moment.

Version numbers for docker image

Need to have access in codebuild to the commit id from codepipeline to tag the image properly. Currently this isn't exposed...not sure how to address? One option is to use CODEBUILD_* env variables to find the current codepipeline execution and then run aws cli to query details of the pipeline execution and get revisionId from there.

Show services

As a developer, i want to be able to show the details of a service so that I can know which version is deployed in which environments.

AC:

• Running mu svc show <svc_name> will display the list of which environments the service is currently running in

Create sample plugin for Spring Boot

Create pipeline (CodePipeline and CodeBuild)

As a microservice developer, i want to be able to create a pipeline for my service to i can continuously deliver my service into production.

AC:

• Run mu pipeline up with a mu file to have a pipeline created
• Will prompt for a GitHub token if none was provided via -t <token
• Will create a CodePipeline with 4 stages: source, build, acceptance, production
• Will create a manual blocker action in Production stage
• Will create a CodeBuild job to compile the source
• Will create a CodeBuild job to create the docker image
• Will create CodeBuild jobs to deploy to acpt and prod

Add support for HTTPS and DNS

• Allow specifying hostedzone in mu.yml to have a resource record added to that zone for the ALB
• Default the hostname in the hostedzone to the env name
• Allow override hostname via name in the mu.yml
• Allow providing certificate in mu.yml to be the ARN of the cert to reference from the ELB. will create ELB listeners for 443 if defined.

UI for environments, services, and pipelines APIs

Service Log Aggregation

Aggregate container logs to CloudWatch logs. Create log group for each service and log stream for each container

Cleanup VPC reference

• Use Fn::Split
• Consider reference Subnet/VPC direct rather than via Import

Blog post on creating a mu plugin

APIs for list environments

Blog post announcing mu

Show pipeline details

Documentation for installing/using UI

What language to use for developing the tool?

Language - TBD. Need to determine the language to use for developing the tool. Options in order of preference include Go, Node.js, Python.

View build logs

Create a subcommand mu pipeline logs [-f] to view the logs for the recent build and -f to follow the logs.

Load plugins from mu.yml

Create mu subcommand to install the UI into an account (S3, API GW, Lambda, Cognito)

Add "why run" support

Preview CFN changeset for the change

VPC enhancements

• ECS subnets are private
• ELB subnets are private for "internal" and public for "internet-facing"
• Allow option for ELB to be "internal" (default: false)
• ECS container instances use private subnets
• Create NAT for managed VPC
• Allow optional "proxy" for ECS container instances

Documentation for developing plugins for mu

Describe range of plugin variations

Why do we need plugins?

What are example plugins?

What capabilities will plugins be able to provide?

Create service (ECS task, ECS service, ALB target group)

As a developer, i want to be able to create with mu, so that i can run my application.

AC:

• Should upsert ECR repo, build docker image and push to ECR with mu service push
• Will allow override of repo to push to by setting imageRepository in mu.yml. If set, ECR repo won't be created
• Will allow override of path to Dockerfile
• Will default image name to the directory that mu.yml is in, can be overridden in mu.yml
• Will default image tag to the short commit id for current git repo, but can be overriden with -t flag
• Should deploy service to ECS environment with mu service deploy <env_name>
• Will allow specifying container port and path patterns for configuration of the ALB target group
• Allow specifying the cpu and memory reservations
• Allow specifying the desired # containers
• Allow specifying the health endpoint for ALB

Capture the "pain" that this tool relieves

Is there a clear example of the pain people suffer without Mu? Is there a way to describe the cost/pain/suffering/errors one suffers when one has to work in a world without Mu? If we have that it would go a long to way to motivate and encourage support.

Create environment (VPC, ECS cluster, ASG/LC for ECS instances, ALB)

As a developer, i want to be able to create my environment with mu, so that i can run my application.

AC:

• Should create new environment stack by running mu env up <env_name>
• Should update (with message of no change) if rerun mu env up <env_name>
• Should require the environment with matching name exists in mu.yml
• Will create VPC stack if no vpcTarget is specified
• If vpcTarget is specified, then the ECS cluster and ALB will be created in the referenced VPC
• Cluster stack will define an ECS cluster, ASG/LC for ECS cluster instances, with autoscale based on memory, and ALB
• VPC stack will define IGW, Subnets, NACL, SG

Add extension point for pipelines