samkennerly / ec2box

Example Terraform module for using Amazon EC2. Each instance includes its own keypair, IAM role, security group, CloudWatch log group, and cloud-init template.

ec2box

There is no cloud. It's just someone else's computer.

abstract

Use Terraform to automatically launch and configure Amazon EC2 resources. Each box includes its own:

• EC2 instance to run programs
• keypair to allow remote SSH login
• security group to control network access
• CloudWatch log group to store and read log messages
• IAM role, policy, and profile to authorize use of other AWS resources
• cloud-init template to configure logs, install software, and run a script

The test module launches example free-tier Ubuntu boxes:

box name	script language	what it does
dorothy	ruby	print timestamped messages
leeroy	bash	print a message, then crash

basics

• Create a new repo from this template.
• Open a terminal and cd to this folder.

authorize AWS

Terraform needs access to AWS security credentials. These can be hard-coded in .tf files, but storing secrets in code can be dangerous. It is safer to use environment variables:

AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY

or credentials files in the user's home folder:

~/.aws/config
~/.aws/credentials

choose a keypair

• Run bin/keygen to generate a keypair.

The private and public keys will be saved here:

etc/ec2box_rsa
etc/ec2box_rsa.pub

When a new box is created, or the public key changes, Terraform will upload a copy of the public key to AWS. Anyone with both keys (public and private) can then login remotely to each box with SSH.

Caution: The private key in the ec2box_rsa file must be kept secret. This repo gitignores it.

launch boxes

• Edit terraform.tfvars to choose an AWS profile and AWS region.
• Run bin/up test to launch all example boxes.

Terraform will save state files when the test module is initialized:

terraform.tfstate
terraform.tfstate.backup

Caution: State files (including remote state) can contain secrets! This repo gitignores them.

inspect boxes

• Run bin/login to login to a box remotely with SSH.

Usually, remote login is unnecessary because boxes can be monitored by reading log messages. Each box creates its own CloudWatch log group and streams logs to it. The pipeline works like this:

1. cloud-init downloads, installs, configures, and starts an AWS CloudWatch agent.
2. The agent creates a log stream and begins streaming from /var/log/syslog.
3. The launch script prints errors to STDERR and all other messages to STDOUT.
4. The shell redirects STDERR and STDOUT to the Ubuntu logger.
5. The logger saves logs to /var/log/syslog.

Cloud-init, system, and launch script logs will then be visible in the AWS CloudWatch console.

deactivate boxes

1. Run bin/down test to destroy all example resources.

contents

Configuration files for the Terraform root module:

• main.tf declares AWS resources to acquire.
• outputs.tf declares outputs to return to other modules.
• variables.tf declares required inputs and their default values.
• terraform.tfvars file sets root module variables.

bin

Short scripts to run Terraform commands:

• bin/clean [FOLDER] autoformats and validates Terraform code.
• bin/down [FOLDER] destroys all resources declared in a folder.
• bin/keygen generates and saves an RSA keypair to the etc folder.
• bin/login [BOXNAME] uses SSH to login to an EC2 instance remotely.
• bin/up [FOLDER] creates or updates all resources declared in a folder.

etc

Default configuration files for each newly-created box:

• ec2box_rsa is an RSA private key.
• ec2box_rsa.pub is an RSA public key.
• install is a script which installs software.
• launch runs in the background when a box is ready to use.
• policy.json is an IAM policy which grants AWS permissions to a box.
• template is a template file for a cloud-init script.

test

Example resources for testing ec2box:

• main.tf declares boxes to be created by terraform apply test.
• outputs.tf declares outputs to be shown by terraform output.
• variables.tf declares inputs to be read from terraform.tfvars.

test/dorothy

Configuration files for a test box named dorothy which prints timestamped messages every 1 second.

dependencies

1. AWS credentials
2. Terraform >= 0.12
3. OpenSSH to run bin/keygen and bin/login
4. jq to run bin/login

examples

Create or update all test boxes. (Terraform will prompt for confirmation.)

> bin/up
Initialize and update test resources
Initializing modules...

...

Apply complete! Resources: 0 added, 0 changed, 0 destroyed.

...

Remote login to dorothy via SSH. (You might need to confirm the host public key.)

> bin/login dorothy
SSH into dorothy at [email protected]
Welcome to Ubuntu 18.04.3 LTS (GNU/Linux 4.15.0-1051-aws x86_64)

...

ubuntu@ip-123-45-67-89:~$

Use SSH to run the launch script on the leeroy box again:

> bin/login leeroy '~/launch'
SSH into leeroy at [email protected]
LEEROOOOOOOOOOOOOOOOOOOOOOOY JENKINS

Destroy all test boxes. (Terraform will prompt for confirmation.)

Destroy all Terraform-managed resources in test

...

Destroy complete! Resources: 14 destroyed.

faq

How do I define my own boxes?

Edit the test folder. Rename it if you want to.

Do I need to use remote state?

No, but it's usually safer than keeping local state files on one person's laptop.

How do I choose different install and/or launch scripts?

See the dorothy box in test/main.tf for an example.

How do I deploy code to a box?

Here are some common methods:

• Upload code to a private S3 bucket and aws s3 sync it to a box.
• Use deploy keys to pull from a private GitHub repository.
• Use AWS CodeDeploy for everything.

To deploy code automatically when a box is created, edit its install script.

Where are the official docs?

• Terraform docs
• Terraform AWS provider docs
• Terraform AWS example modules