cloudtools / stacker

An AWS CloudFormation Stack orchestrator/manager.

stacker

For full documentation, please see the readthedocs site.

Click here to join the Slack team for stacker, and then join the #stacker channel!

About

stacker is a tool and library used to create & update multiple CloudFormation stacks. It was originally written at Remind and released to the open source community.

stacker Blueprints are written in troposphere, though the purpose of most templates is to keep them as generic as possible and then use configuration to modify them.

At Remind we use stacker to manage all of our Cloudformation stacks - both in development, staging, and production without any major issues.

Requirements

• Python 3.7+

Stacker Command

The stacker command has sub-commands, similar to git.

Here are some examples:

build:

handles taking your stack config and then launching or updating stacks as necessary.

destroy:

tears down your stacks

diff:

compares your currently deployed stack templates to your config files

info:

prints information about your currently deployed stacks

We document these sub-commands in full along with others, in the documentation.

Getting Started

stacker_cookiecutter: https://github.com/cloudtools/stacker_cookiecutter

We recommend creating your base stacker project using stacker_cookiecutter. This tool will install all the needed dependencies and created the project directory structure and files. The resulting files are well documented with comments to explain their purpose and examples on how to extend.

stacker_blueprints: https://github.com/cloudtools/stacker_blueprints

This repository holds working examples of stacker blueprints. Each blueprint works in isolation and may be referenced, extended, or copied into your project files. The blueprints are written in Python and use the troposphere library.

stacker reference documentation:

We document all functionality and features of stacker in our extensive reference documentation located at readthedocs.

AWS OSS Blog: https://aws.amazon.com/blogs/opensource/using-aws-codepipeline-and-open-source-tools-for-at-scale-infrastructure-deployment/

The AWS OSS Blog has a getting started guide using stacker with AWS CodePipeline.

Docker

Stack can also be executed from Docker. Use this method to run stacker if you want to avoid setting up a python environment:

docker run -it -v `pwd`:/stacks remind101/stacker build ...

stacker's Issues

Compatibility with AWS Multi-Factor Auth

I see mention of env variables for AWS_ACCESS_KEY_ID & AWS_SECRET_ACCESS_KEY, but how about for when AWS_SESSION_TOKEN exists when using MFA?

I couldn't find anywhere in the code where these vars were explicitly referenced, so the standard boto auth mechanism should work, but it doesn't seem to from our tests.

<ErrorResponse xmlns="https://route53.amazonaws.com/doc/2013-04-01/"><Error><Type>Sender</Type><Code>InvalidClientTokenId</Code><Message>The security token included in the request is invalid</Message></Error><RequestId>cdb0b289-67e1-11e5-90fe-17e65d18e686</RequestId></ErrorResponse>

require_stacks tag can get beyond AWS limit for tag value length

When dealing with a large(r) number of stacks, the tag value can grow beyond the AWS limits (particularly the require_stacks tag). This results in an error in deployment.

I noticed that there is another issue regarding tags (#44), and a comment that the tags aren't actually used for anything. Should the tags be nuked (kill two birds)? I'd be happy to put a pull request together, but I'm not sure what the right approach is.

Hi Mike :).

VPC blueprint adds BaseDomain (external) before InternalDomain to search path

This seems like it'd almost always be wrong:

https://github.com/remind101/stacker/blob/master/stacker/blueprints/vpc.py#L127

Should change the ordering of them.

Launch/modify subset of stacks

Right now you can modify the config, but that's messy. It'd be cool if you could specify a list of stacks in the config to modify.

Changing the name of an output doesn't update the stack

If I change the name of an output, it won't get updated until I change something else with the template.

Is this an issue with CloudFormation?

ie.

if I change an output from "EmpireControllerSG" to "EmpireControllerSecurityGroup" (because being explicit is always better ;)) and run build, stacker will skip updating that stack because it says nothing has changed and then the renamed variable won't be accessible.

README.rst needs new subcommand syntax

@mhahn - not sure if you already have this in another of your PRs?

Allow multiple configs?

An idea - maybe allow multiple configs, so you can have your 'general config' and then more specific overrides for environments, etc.

Issue force updating new RDS Db

CloudFormation cannot update a stack when a custom-named resource requires replacing. Rename apirdsdb and update the stack again.

i know its rare that you want to update a db and replace it, but in this case i did. not sure if we care about handling it.

VPC creation error when creating a stack in us-west-2

I used the standard conf/empire/empire.yaml file and launched a stack with the following command:

stacker -e example.env -r us-west-2 brianz.bz empire.yaml

The VPC failed to create with the following error:

CREATE_FAILED   AWS::EC2::Subnet    PrivateSubnet3  Template error: Fn::Select cannot select nonexistent value at index 3

modify stacker_blueprints to use vpc nat gateway instead of nat instances

Since cloudformation and troposphere support nat gateways now, should they be used instead of nat instances? I'm working on trying to modify vpc.py to support that, but I keep doing something wrong, so I don't have a pull request ready yet. Is this a direction in which you are moving, or are you planning on sticking with nat instances?

Also, thanks for doing some great work with this and empire!

Support change sets

AWS recently announced support for change sets in CloudFormation.

It would be pretty awesome to support change sets inside stacker, so you can preview changes before applying them.

'reason' for an item being skipped in checkpoint would be useful

We have a few reasons a stack might get skipped - it'd be good if in the checkpoint output, as well as other places, that reason was shared.

Support for stack policies

CloudFormation has support for setting stack policies to prevent updates to protected resources. We should support this within stacker.

Support extending ASG blueprint

Did some of this in:

#100

but wanted to get @phobologic thoughts from the RDS refactor about any other things we should do if we refactor these.

If a stack already exists, and a parameter isn't provided on an update, pull from existing parameters

CloudFormation's webui already does this for you, and it's convenient and helps prevent accidentally changing a parameter that you don't want to update.

Maybe make it so you have to provide an additional flag to actually change a parameter on an update? That may be a bit extreme and annoying.

Support "pre_build" and "post_build" hooks for individual stacks

Right now if a stack depends on a pre_build hook, we put it at the top of the conf file. This makes it hard to tell which stacks depend on which hooks.

I'd rather have something like:

stacks:
  - name: vpc
    ...
  - name: ecs:
    class_path: blueprints.ecs.DefaultEcs
    pre_build:
      - path: stacker.hooks.iam.create_ecs_service_role
        required: true

then if I remove the ecs stack, the hook that it depends on is removed.

this also would support my usecase with vault, where i'm going to have dependent stacks depend on referencing values from the vault instance. on initial launch, i'm going to need to pause post building the vault instances, configure vault, then continue.

Empire stack - Authentication to docker hub fails when $ is present in password

If you have a $ in your password, the auth token generated for /etc/empire/dockercfg is incorrect. This was verified by doing:

make_dockercfg uname crazypa$word [email protected]

...and noting that the auth token is different than one which is known to be good.

Escaping the $ produces the correct token:

make_dockercfg uname crazypa\$word [email protected]

remove aws_helper

aws_helper isn't used in many places any longer - only in the hooks, and there's really no reason for it.

Need the ability to delete() the stacks

stacker should figure out how to delete things in order, based on dependencies.

If a stack launch fails and the stack is 'locked' stacker will try to update

[2015-12-03T17:51:43] Launching stack example-com-vpc now.
[2015-12-03T17:51:43] Attempting to update stack example-com-vpc.
[2015-12-03T17:51:50] Launching stack example-com-MasterDB now.
[2015-12-03T17:51:50] Stack example-com-MasterDB not found, creating.
[2015-12-03T17:51:51] Launching stack example-com-empireDB now.
[2015-12-03T17:51:51] Stack example-com-empireDB not found, creating.
[2015-12-03T17:52:30] Launching stack example-com-MasterDB now.
[2015-12-03T17:52:30] Stack MasterDB locked and not in --force list. Refusing to update.

What happened there was that the MasterDB was launched, but then there was an issue with the creation of the stack, so CloudFormation went into a rollback. Usually we'd crash immediately the next time we checked the stack, but in this case it didn't crash and instead thought it should update the stack.

I think this is due to the fact that we check for the lock before trying the update, which when it sees its locked it skips the update, which is where the usual exception comes from.

Not dangerous at this point, just confusing.

See: https://github.com/remind101/stacker/blob/master/stacker/actions/build.py#L185

Excessive Memory Consumption

As my stacker config grows, the memory consumption has grown significantly. I have a vagrant development environment that is running out of memory when attempting to deploy a stack. I've had to increase my vagrant box memory to 4gb in order to handle a stacker deployment with 17 stacks.

preview mode

It'd be good to have a mode where stacker figures out what would change and lets you know.

Use default region for s3

If you switch regions but use the same namespace, you'll get conflicts.

Running into error when tailing

keep getting:

ssl.SSLError: [Errno 1] _ssl.c:1429: error:1408F119:SSL routines:SSL3_GET_RECORD:decryption failed or bad record mac

Boolean parameters should be converted to strings

true in YAML converts to python's True boolean. Parameters are never booleans, and the strings true and false are somewhat common as strings in Parameters. I can't think of a reason to leave them as the python booleans, as right now to use true for example you have to use !!str true

inconsistent postgres names

https://github.com/remind101/stacker/blob/master/stacker/blueprints/postgres.py#L9
https://github.com/remind101/stacker/blob/master/stacker/blueprints/postgres.py#L11

create something like:

PostgresRDSapiRdsDb
RdsSGapiRdsDb

the name should always go in the front to stay consistent with other camel cased names.

Look into using cloudformation tags again

We removed them with #44 - now CF does provide a method for updating tags, so we should look into whether or not they'd be useful again.

Allow specifying stack name for parameters on the command line

Make it so --parameter <stack_name>::<parameter_name> applies that parameter only to that stack. Right now they are global.

Supporting eu-west-1 in the empire build

I've done the work locally to support eu-west-1 for the stacker empire build but that involved reducing the number of AZs from 4 to 3 (as eu-west-1 only has 3).

I'd like to give the updates back to you, but is there a way to add conditional logic into the conf/empire/empire.yaml file to allow 4 everywhere but in eu-west-1?

Need to handle ROLLBACK messages better

While rolling out an RDS stack had an issue - it rolled back, but it totally crashed the builder:

Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "build/bdist.macosx-10.10-intel/egg/stacker/builder.py", line 349, in build
    self.launch_stack(stack_name, template)
  File "build/bdist.macosx-10.10-intel/egg/stacker/builder.py", line 276, in launch_stack
    capabilities=['CAPABILITY_IAM'])
  File "/Users/mike/venv/stacker/lib/python2.7/site-packages/boto/cloudformation/connection.py", line 527, in update_stack
    body = self._do_request('UpdateStack', params, '/', 'POST')
  File "/Users/mike/venv/stacker/lib/python2.7/site-packages/boto/cloudformation/connection.py", line 295, in _do_request
    raise self.ResponseError(response.status, response.reason, body=body)
BotoServerError: BotoServerError: 400 Bad Request
{"Error":{"Code":"ValidationError","Message":"Stack:arn:aws:cloudformation:us-east-1:427009882155:stack/mike-remind-com-empireDB/6fdbeba0-c2cf-11e4-9ca6-500150b34c44 is in ROLLBACK_COMPLETE state and can not be updated.","Type":"Sender"},"RequestId":"86377e96-c2cf-11e4-839d-5f31d7b681dd"}

Non-us-east stacks have issue w/ create_bucket

http://boto.readthedocs.org/en/latest/ref/s3.html#boto.s3.connection.S3Connection.create_bucket

Basically anything that's not in us-east-1 needs to have Location set to the appropriate region.

Add a --dryrun flag

Sometimes, I just want all the templates to be generated as a way of syntax checking, which would provide faster feedback than deploying (and would also make it useful as a test in CI).

Run in existing environment

The current example (in conf/example.yaml) seems to assume that I will set up everything using stacker. Do you have any examples of a stack for an existing environment? For example, if I already have a VPC configured, and want to add a subnet using stacker, how would I specify the VPC ID?

Do let me know if there's a better place to ask these kinds of questions (irc/slack/mailinglist).

Support programatically constructing outputs

I've been thinking about a way we could make stacker -> empire conf simpler.

Right now I setup the stacks, then have to examine the outputs (mostly just DB addresses) and map them to DATABASE_URL and REDIS_URL values in an empire conf. I think something like this could be useful:

  - name: apiRdsDb
    class_path: stacker.blueprints.rds.postgres.MasterInstance
    locked: true
    parameters:
      << : *vpc_parameters
      ... db parameters ...
    outputs:
      DATABASE_URL:
        - path: stacker.outputs.construct_database_url
          args:
            username: ${api_rds_master_user}
            password: ${api_rds_master_user_password}
            hostname: apiRdsDb::DBAddress
            datbase: ${api_rds_db_name}

then maybe some flag to stacker info could output all of the "outputs" you've defined in the config, ie.

DATABASE_URL=postgres://user:password@hostname/dbname

that could be appended to an existing env file or be the start of an env file you load within empire via emp env-load

Started seeing issues with TCS with new amis

aws/amazon-ecs-agent#102

Combine namespace & environment

Right now the namespace & environment seem excessive together. Spoke with a few people about this and the current proposal is to combine them - basically do away with the namespace on the command line, and instead require an environment file with each run. In the environment file, the minimum contents would be something like:

namespace: <namespace>

Command line & config need cleanup

There's a lot of duplication, and honestly it's gotten a little away from me.

update_stack does not modify tags

boto/boto#2032

Looks like this isn't even possible in the Cloudformation API, which is frustrating. It basically means that we can't trust the tags of a stack - and certainly can't rely on them for anything.

Fortunately we don't currently rely on the tags for anything, but it was planned to use the dependent stack tag when writing code to destroy stacks. Frustrating.

Make namespace optional?

If you're using separate AWS accounts for each "meta" stack, having a namespace doesn't add much value, and actually makes it harder to search through the cloudformation generated names for things.

What do you think about making the namespace optional and if not provided, just upload the templates direct to cloudformation instead of s3 to avoid having to create the s3 bucket before hand?

RFC: blueprint acquisition system

Right now it's kind of annoying to use third party blueprints in stacker - you have to get them (git pull), then install them in your python path (python setup.py install), only then can you use them.

I'm trying to think of a better way to do this, so opening this up for discussion from 'the community' :)

DNS Stuff

Should automatically setup a subdomain for the user when it is used, and hopefully add things to that domain.

Will likely need a DNS service that updates Route53 based on hosts coming up. Maybe tie that to etcd for incrementing names?

Empire AMI seems to be targetting old ECS agent (at least for eu-west-1)

It seems like the latest AMI ami-3cef374f is targetting ECS agent version 1.4.0 which has some really bad disconnection issues.

Could you walk me through how to update the AMI?

Add ability to introspect templates from the CLI

For example, it'd be good if you could get a list of expected Parameters & Outputs for a given stack, like VPC.

Fn::GetAZs only returns VPC's with default subnets

I don't think this is something that stacker needs to try to handle, but wanted to put it here in case anyone runs into it.

Within the vpc stack we iterate through AZCount and select from Fn::GetAzs. However, it's possible that if you ever delete the default subnets in an aws account that this function will not return anything:

From http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/intrinsic-function-reference-getavailabilityzones.html

For the EC2-Classic platform, the Fn::GetAZs function returns all Availability Zones for a region. For the EC2-VPC platform, the Fn::GetAZs function returns only Availability Zones that have a default subnet unless none of the Availability Zones has a default subnet; in that case, all Availability Zones are returned.

AWS throttling API responses should be retried

Stacker exits upon hitting an AWS throttling response. I would expect the request to be retried (perhaps after a brief back-off):

Traceback (most recent call last):
  File "/usr/local/bin/stacker", line 145, in <module>
    builder.build(stack_definitions)
  File "/usr/local/lib/python2.7/dist-packages/stacker/builder.py", line 368, in build
    self.launch_stack(stack_name, blueprint)
  File "/usr/local/lib/python2.7/dist-packages/stacker/builder.py", line 264, in launch_stack
    stack = self.get_stack(full_name)
  File "/usr/local/lib/python2.7/dist-packages/stacker/builder.py", line 212, in get_stack
    return self.conn.cloudformation.describe_stacks(stack_full_name)[0]
  File "/usr/lib/python2.7/dist-packages/boto/cloudformation/connection.py", line 655, in describe_stacks
    return self.get_list('DescribeStacks', params, [('member', Stack)])
  File "/usr/lib/python2.7/dist-packages/boto/connection.py", line 1182, in get_list
    raise self.ResponseError(response.status, response.reason, body)
boto.exception.BotoServerError: BotoServerError: 400 Bad Request
<ErrorResponse xmlns="http://cloudformation.amazonaws.com/doc/2010-05-15/">
  <Error>
    <Type>Sender</Type>
    <Code>Throttling</Code>
    <Message>Rate exceeded</Message>
  </Error>
  <RequestId>f9428caa-39ff-11e5-bb46-05293e3ea2eb</RequestId>
</ErrorResponse>

handle multiple users updating a stack at once

Right now there's no special logic that would stop 2 people from attempting to update stacks at the same time. In general the second attempt will see the stacks being updated, and probably do the right thing, but I can see some cases where that wouldn't happen and could leave stacks in invalid states.

So - probably worth having some way to lock the stack while you're working on it. Not sure where that would be handled.

Check all required Parameters

Stacker should be able to verify that all necessary Parameters are provided before attempting to launch any stacks.

Support stack policies

user-data in empire stacks is horrible

Need a better way to do this - especially because of credentials being shared in the stack.

blueprint versioning

Blueprints change rather often - there are two major ways they change that we should care about:

• the resources the blueprint produces change
• the blueprint class itself somehow changes, like if we add/remove arguments in init

In case 1 it'd be good to make sure that if there is a potentially dangerous change that would require the destruction of resources that we let people know that.

In case 2 - well, we need to make sure that the blueprints match their expected behavior in stacker.

Not sure how to handle this.