andrew-the-drawer / ops-compilation

Compilation of infrastructure/dev-ops stacks and modules

Ops compilation

Introduction

This is my devops/infrastructure stack-and-module compilation that is easy to set-up for multiple projects from MVPs to large-scale types.

Tech stacks:

• Terraform
• Serverless framework (AWS cloudformation)
• Ansible
• Docker & Docker swarm
• Scripting languages (Python, NodeJS, Bash,...)
• CI frameworks (Github actions, CircleCI,...)

Structure:

• /core: Contains core setup files (ansible playbooks/roles, terraform modules/stacks, CI templates,...).
• /config: Contains templated configuration files for each infrastructure stacks, that could be customized depends on each project needs (Dockerfile, docker-compose.yaml, yaml config file,...)
• /resources: Extra resources/documentation that is attached for some infrastructure stacks/modules

Notes:

If you guys want to contribute, please open a PR from the master branch, I'll review it if possible 😀, though I'm quite busy.

Also, any suggestions are welcome by opening new issues in Github, I'll try to discuss on the feature/bug-fixes or answering the questions from you guys.

Thanks again for noticing. Please leave a star if you find this repo useful.

ops-compilation's Issues

Tryout CI/CD pipeline with aws-cdk or terraform-cdk (for ECS service deployment)

https://github.com/hashicorp/terraform-cdk

https://www.npmjs.com/package/aws-cdk

I'm more familiar with JS/TS stack, but you guys can try with other programming language as well. Super welcome!

Checklist (More to be added as a separate issue):

• ECS service deployment

AWS ECS service deployment with Terraform

• External scripts to build a ECS task definition
• Modules to build & deploy ECS service config

Add CI/CT for ansible

• Lint?
• Maintainable ansible configuration?
• Test and CI integration

Docker swarm on EC2 auto-scaling group

Set up:

• 1 node, which will be manager node
• 1 ASG => worker nodes

Add auto-scaling for docker swarm service on EC2

Since docker swarm doesn't support auto-scaling for services, we need to add ad-hoc system to support auto-scaling

Suggestion: Add monitoring tools for docker swarm --> event trigger for certain thresholds --> Run a command to scale up a system

Bonus:

• Delay/cool-down period
• Custom metrics support

Cronjob managment service in docker swarm

Deploy cron using crontab in manager mode, which will launch ephemeral service https://blog.alexellis.io/containers-on-swarm/

Use Docker config for saving env variables/secrets in Docker swarm

Add support for rolling updates in docker swarm service

Basic rolling update feature:

• Replace service task 1 by 1 (schedule new task first, delete the old task later)
• Support rollback if failed somehow
• Interface to notify if the deployment fails or succeeds

Add ansible for PostgreSQL setup

Add a PostgreSQL setup by ansible:

• Deploy by docker
• Single mode/cluster mode
• Backup/restore cronjob using S3 as a object bucket

Can choose to deploy on any remote environment (EC2,...)

Use python virtual environment to install dependencies

Some Ansible packages/plugins required Python dependencies with different version (boto3, botocore,....). Should be able auto-install using python virtual env

Use Hashicorp Packer to set up AMI of CentOS + Docker

https://www.packer.io/

Subtask of #10

Research + breakdown the infrastructure for GraphQL

• Normal HTTP requests
• Subscription using web-socket
• Centralized customized GraphQL engine/service (Boilerplate)
• Attribute/role-based access control

Set up Docusaurus

https://docusaurus.io/

Deploy to S3 and setup as a static website