balduran / awesome-ecs

A curated list of awesome ECS guides, development tools, and resources

A curated list of guides, development tools, and resources for Amazon Elastic Container Service (ECS). This list includes both community created content as well as content created by AWS.

Want to add something? Open a PR! 🙂

Pick your container hosting strategy:

• AWS Fargate - AWS Fargate is a technology for Amazon ECS that allows you to run containers without having to manage servers or clusters.
• Self hosted in EC2 - Running your own cluster of EC2 instances to host your containers gives you the most control over price (ability to run on spot instances or reserved instances) as well as configuration.

AWS Fargate

Setting up Fargate

• fargate - Command line tool for interacting with AWS Fargate. With just a single command you can build, push, and launch your container in Fargate, orchestrated by ECS.
• CloudFormation Templates - Sample CloudFormation templates to help you spin up a Fargate cluster, and service in that cluster automatically
• Terraform - Use Terraform to deploy your docker containers in Fargate
• Terraform Airship Modules Full getting started guide to setup your Fargate ECS Services.
• fargate-create - A CLI tool for creating new projects based on Terraform templates and Fargate CLI. Supported stacks:
  • Web Application (ALB - HTTP/HTTPS)
  • Network Application (NLB)
  • Background Worker (Service)
  • Scheduled Task (Job)

Application Deployments in Fargate

• Deploying a Rails app to Fargate - Step by step walkthrough of deploying a Ruby + RDS app, with helpful debugging tips
• How to use AWS Fargate and Lambda for long-running processes in a Serverless app - Great tutorial showing how to leverage the power of long running docker containers in Fargate alongside Lambda. Example application processes video files to extract thumbnails using just S3, Lambda, and Fargate... no EC2.
• Microservice CI/CD to AWS Fargate - Learn how to use Stelligent Mu to setup a CI/CD pipeline for deploying a 3 tier, polyglot, microservice based application to AWS Fargate
• Wonqa is a tool for spinning up disposable QA environments in AWS Fargate, with SSL enabled by Let's Encrypt. More details about Wonqa on the Wonder Engineering blog.
• Setting up service discovery for AWS Fargate using CloudFormation
• Using the Clair image scanner with Fargate, ECR and CodeBuild/CodePipeline

Self hosted in EC2

• Getting Started
• Infrastructure as Code
• Build and Deploy Tools
• Implementation Guides
• Open Source
• Reference Architectures
• Continuous Integration / Continuous Deployment
• Presentations
• Tech Blogs

Getting Started

• The Hitchhikers Guide to AWS ECS and Docker by J. Cole Morrison - Introduction to AWS ECS concepts
• Working with AWS ECS by Sriram Rajan of Rackspace

Infrastructure as Code

Examples of using tools to describe your ECS infrastructure as code, for automation of deployments:

• mu - Automates everything relating to ECS devops and CI/CD. This framework lets you write a simple metadata file and it constructs all the infrastructure you need so that you can deploy to ECS by simply pushing to your Git repo.
• Sample CloudFormation templates for ECS - Examples of launching containers with both public, and private networking, behind a public facing load balancer, as well as behind a private, internal load balancer.
• CloudFormation ECS - Reference architecture for deploying microservices to ECS in tiered VPC with NAT gateways and two availability zones.
• Terraform ECS by Armin Coralic - Production ready AWS ECS infrastructure as code with Terraform
• Airship Terraform ECS by Maarten van der Hoef et al. - Multiple highly documented ECS Modules for deploying ECS Clusters and Services with Terraform
• CloudFormation Templates by Cloudonaut
• ecsq - A developer friendly tool for querying the state of an ECS cluster
• deployfish - Write a simplified deployfish.yml file describing your deployment and let this tool handle the heavy lifting of deploying your service.

Build and Deploy Tools

Tools to help you interact with ECS to launch your containers on your cluster of self managed EC2 instances:

• coldbrew - Fantastic tool that provisions ECS infrastructure, builds and deploys your container, and connects your services to an application load balancer automatically. Has a great developer experience for day to day use
• ecs-cli - Docker Compose compatible deployment tool by AWS
• empire - Control layer on top of ECS that provides a Heroku like workflow
• broadside - Ruby based command line tool for deploying to ECS
• UFO - Ruby based tool for building containers and shipping them to ECS
• bash deployment script by Justin Kulesza
• pnzr - Go based tool for building and pushing to ECS, also has integraton with AWS KMS for secrets management.
• deplojo - Python based deployment tool using ECS
• convox - Easily build, deploy and scale applications on ECS
• ecsctl - Open source tool similar to Kubernetes kubectl for ECS.
• ecs-deploy - Simple but powerful tool for initiating automatic blue green deploys on ECS
• ecspresso - Minimalistic: JSON file goes in, service launches
• shipctl - Tool that supports deploying a task on ECS, rolling back, or just running a one-off task
• ecsdeploy - A client and simplified web interface for managing your ECS cluster, rolling out and rolling back application versions
• ecs-service - CLI tool for deploying to ECS using CloudFormation with support for .env files for environment specific configuration of your containers
• kms-env - CLI tool for managing secrets using AWS KMS in .env files which can be used in conjunction with ecs-service to supply secrets to your containers
• Chaos Toolkit - Chaos toolkit supports ECS as a target for chaos engineering

Implementation Guides

Examples of how to do advanced customizations on your ECS cluster:

• Autoscaling
  • Autoscaling services in ECS - How to autoscale the number of service tasks
  • Autoscaling the cluster in ECS - How to autoscale the number of EC2 instances in a cluster
  • Advanced ECS cluster autoscaling - A more advanced technique for autoscaling an ECS cluster, with more effective scale in as well as out
  • Lambda based approach for cluster scale-in
  • Extending ECS autoscaling for under $2 a month with Lambda
  • Scale in script in Python
• Networking
  • Private subnets tutorial
  • End-to-end TLS traffic - How to setup end-to-end TLS from client to container, as well as from container to container
• Service Discovery
  • Using built-in Route53 service discovery
  • Using DNS
  • Using Consul
  • Using Weaveworks
  • Using linkerd
  • Using HAProxy
• Data Persistance
  • Using Amazon Elastic File System
• Secrets Management
  • Using Parameter Store and IAM Roles for Tasks
  • Using ECS Task roles for managing AWS credentials
  • The right way to store secrets using Parameter Store
  • Open source tool for automatically getting secrets from SSM into your docker environment
• Administration
  • Automatically draining cluster instances before termination
  • Monitor cluster state with Cloudwatch event stream
  • React to ECS events such as crashed containers using a serverless function
  • Run an ECS Task on every cluster instance
• Docker for .NET Developers
  • Part One: Intro
  • Part Two: Our First dockerfile

Open Source

• Blox - Framework for advanced cluster management and scheduling
• Watchbot - This tool by Mapbox helps you run data processing across an ECS cluster in response to external events
• ecs-export - A tool for exporting ECS cluster metrics to Prometheus for advanced querying
• docker-elk-ecs - Connecting Amazon ECS container logs to an ELK (Elasticsearch, Logstash, Kibana) stack
• Sample task definitions - Sample task definitions for running applications like Nginx, Tomcat, Gunicorn, Wildfly, Kibana, and Jetty as containers under Amazon ECS

Reference Architectures

• Microservice cloudformation stack
• Sock shop microservices demo on Amazon ECS
• Node.js Microservices
• Java Microservices on AWS ECS
• Swift ECS Workshop
• NGINX Reverse Proxy sidecar container on AWS ECS
• Deploying a Deep Learning Framework on ECS
• Powering your Amazon ECS Cluster with Amazon EC2 Spot Instances
• Reactive Microservices Architectures with Amazon ECS, AWS Lambda, Amazon Kinesis Streams, Amazon ElastiCache, and Amazon DynamoDB
• Cats n' Dogs - A fun workshop that covers service and container-instance auto-scaling, spot-fleet integration, container placement strategies, service discovery, secrets management with AWS Systems Manager Parameter Store, among other things.

Continuous Integration / Continuous Deployment

• CI/CD using:
  • AWS CodePipeline
  • Atlassian Bamboo
  • JetBrains TeamCity
  • Lambda
  • CircleCI
  • Shippable
  • Gitlab
  • SemaphoreCI
• CI/CD mechanics:
  • Rolling blue/green deploy in place
  • Canary Deploy
  • Isolated Regression Testing

Presentations

• Running your Dockerized application(s) on AWS EC2 Container Service by Marco Pas
• Microservices on AWS with Weaveworks
• Running a Virtual World via ECS - Linden Labs on their usage of ECS
• Advanced Task Scheduling with AWS ECS
• Instacart on running microservices on Amazon ECS
• Building Next-Generation Applications with Amazon ECS - How Meteor Built Galaxy on Amazon ECS
• Amazon ECS at Coursera: A General Purpose Microservice

Tech Blogs

• Airtime - Microservice Continuous Integration Made Easy with AWS ECS
• Segment - Rebuilding Our Infrastructure with Docker, ECS, and Terraform
• Nextdoor - How Nextdoor made a 10x improvement in release times with Docker and Amazon ECS
• MapBox - We switched to Amazon ECS and you won't believe what happened next
• Jimdo - Container Based Crons
• Wrapp - On how EC2 Container service simplified our infrastructure stack
• Docker on AWS: from containerization to orchestration
• Deploying Distributed Stateful Applications on ECS - Akka Cluster as an example
• Migration of our video encoder to AWS - How and why Pornhub moved massive amounts of video encoding from Mesos to an autoscaling ECS cluster on AWS
• Realtor - A Better ECS
• Building Blocks of Amazon ECS - Learn about the basic building blocks of ECS and how they fit together to fully understand how it works and how you can use it!

Courses

• Microservices with Docker, Flask, and React - Learn how to build, test, and deploy microservices powered by Docker, Flask, React Amazon ECS!
• ECS Workshop using AWS Fargate and Mu - Learn how to deploy a 3 tier, polyglot, microservice based application to AWS Fargate for ECS using the Mu framework!
• AWS Developer: Optimizing on AWS - Learn how you can optimize your applications on AWS with Amazon ECS!