garyd203 / ssmash

ssmash - Simple application configuration management for AWS SSM Parameter Store

ssmash

ssmash, the SSM AppConfig Storage Helper, is an easy-to-use application configuration management tool for AWS deployments. You specify hierarchical configuration values in a simple YAML file, and ssmash will turn that into an AWS CloudFormation file that stores your configuration values in the SSM Parameter Store.

ssmash is mainly intended for application developers who are at least partly involved in the deployment and operations of their applications. If you want to externalise (some of) the runtime configuration of your application, this is a simple and cheap solution. If you also want to be able to automatically restart your application when it's configuration changes, then this is the tool for you

Installation

Install ssmash using pip, the standard python package installer:

$ pip install ssmash

You will probably use ssmash with the AWS command line tools, so install and configure that too, if you haven't already:

$ pip install awscli
$ aws configure

Example

Suppose you have an input file like this:

acme:
    shipping-labels-service:
        enable-fast-delivery: true
        explosive-purchase-limit: 1000
        greeting: hello world
        whitelist-users:
            - coyote
            - roadrunner

Then run ssmash:

$ ssmash -i acme_prod_config.yaml -o cloud_formation_template.yaml
$ aws cloudformation deploy \
    --stack-name "acme-prod-config" --template-file cloud_formation_template.yaml \
    --no-fail-on-empty-changeset

You will now have the following parameters in AWS Systems Manager, that can be loaded as a string inside your application:

• /acme/shipping-labels-service/enable-fast-delivery = "true"
• /acme/shipping-labels-service/explosive-purchase-limit = "1000"
• /acme/shipping-labels-service/greeting = "hello world"
• /acme/shipping-labels-service/whitelist-users = "coyote,roadrunner"

Automated Application Restarts

Most of the time, your application loads its configuration at startup. Depending on your application, the safest and easiest way to reload its configuration is to simply restart it.

ssmash has built-in support to restart some types of application as part of the deployment process. We do this by telling it to "invalidate" the configuration used by the application.

Docker with AWS ECS

ssmash can generate CloudFormation that will safely restart the Tasks in an ECS Service once your configuration has changed, and make the successful deployment of your new application configuration depend upon the successful restart of that Service. Just specify the target ECS service using extra command line parameters, like so:

$ ssmash -i acme_prod_config.yaml -o cloud_formation_template.yaml \
    invalidate-ecs \
    --cluster-name acme-prod-cluster \
    --service-name shipping-labels-service \
    --role-name arn:aws:iam::123456789012:role/acme-ecs-admin
$ aws cloudformation deploy \
    --stack-name "acme-prod-config" --template-file cloud_formation_template.yaml \
    --no-fail-on-empty-changeset

You can also refer to the name of a CloudFormation Export instead of using the name directly (eg. if your service has a non-obvious generated name), using the interchangeable command line parameters for --cluster-import and --service-import and --role-import.

Serverless with AWS Lambda

ssmash can generate CloudFormation that will safely cause your serverless functions to discard their virtual machine (aka "Execution Context"), meaning they effectively reload their configuration. To access this secret sauce, just add a couple more command line parameters:

$ ssmash -i acme_prod_config.yaml -o cloud_formation_template.yaml \
    invalidate-lambda \
    --function-name shipping-label-printer-function \
    --role-name arn:aws:iam::123456789012:role/acme-serverless-admin
$ aws cloudformation deploy \
    --stack-name "acme-prod-config" --template-file cloud_formation_template.yaml \
    --no-fail-on-empty-changeset

You can also refer to the name of a CloudFormation Export instead of using the name directly, using the interchangeable command line parameters for --function-import and --role-import.

Advanced: Automated Restarts For Only Some Parameters

Automated application restarts are great, but they don't scale when you have a single configuration file that is used by multiple applications - you don't want to restartevery application every time one of the config values changes. Happily, ssmash can handle that too - you just need to invoke the magic (madness!) of YAML tags, which allow us to add metadata to any part of the configuration hierarchy (either leaf configuration values, or tree nodes).

First, let's extend the above example to include configuration for another application:

acme:
    common:
        enable-slapstick: true
        region: us-west-2
    shipping-labels-service:
        enable-fast-delivery: true
        explosive-purchase-limit: 1000
        greeting: hello world
        whitelist-users:
            - coyote
            - roadrunner
    warehouse-service:
        item-substitutes:
            birdseed: "iron pellets"
            parachute: "backpack"

Now we add a special .ssmash-config key to tell ssmash how to restart our applications. Then we annotate the configuration hierarchy using custom YAML tags to tell ssmash which applications are invalidated by which parts of the configuration hierarchy:

---
.ssmash-config:
    invalidations:
        # The dictionary key here ("shipping-labels") is used in the
        # configuration hierarchy to refer to this application
        shipping-labels: !ecs-invalidation
            # The `!ecs-invalidation` tag tells ssmash that this application
            # uses ECS, and the configuration fields correspond to those used
            # on the command line
            cluster_name: acme-prod-cluster
            service_name: shipping-label-service
            role_name: arn:aws:iam::123456789012:role/acme-ecs-admin
        warehousing: !ecs-invalidation
            cluster_name: acme-prod-cluster
            service_name: warehouse-service
            role_name: arn:aws:iam::123456789012:role/acme-ecs-admin
acme:
    common:
        # This is a single leaf configuration value called "enable-slapstick",
        # which will cause both applications to restart when it is changed
        ? !item { invalidates: [ shipping-labels, warehousing ], key: enable-slapstick }
        : true
        region: us-west-2
    # This is a tree node called "shipping-labels-service", which will cause
    # the "shipping-labels" application defined above to restart when any of
    # it's configuration values are changed
    ? !item { invalidates: [ shipping-labels ], key: shipping-labels-service }
    :
        enable-fast-delivery: true
        explosive-purchase-limit: 1000
        greeting: hello world
        whitelist-users:
            - coyote
            - roadrunner
    # This is a tree node called "warehouse-service", which will cause
    # the "warehousing" application defined above to restart when any of
    # it's configuration values are changed
    ? !item { invalidates: [ warehousing ], key: warehouse-service }
    :
        item-substitutes:
            birdseed: "iron pellets"
            parachute: "backpack"

Then run ssmash normally:

$ ssmash -i acme_prod_config.yaml -o cloud_formation_template.yaml
$ aws cloudformation deploy \
    --stack-name "acme-prod-config" --template-file cloud_formation_template.yaml \
    --no-fail-on-empty-changeset

ssmash's Issues

use AGPL

As a user I can specify AWS tags for SSM parameters

Support AWS tags specified at any node in the input configuration, which get applied to that node and lower. Specify them using a custom YAML tag? or perhaps just a well-defined and unusual field name.

Also support tags specified at the command line, to be applied to the entire tree

As a user I can specify parameters with a list value

As a user I can specify parameters with a list value

Add a CLI option for `--version`

Add a CLI option for --version

Invalidation occurs too early, before old parameters are deleted

When an ECS Service (or other dependency, like a Lambda Function) is invalidated after changing the parameters, we implement this in CloudFormation by "updating" a custom resource. In terms of the CloudFormation lifecycle, this means that we create + update SSM parameters, then restart the dependent service, and then delete SSM Parameters that have been removed. Obviously, this means that the service is restarted too early, and will detect the old parameters that should be removed :-(

We can solve this by hooking into the CloudFormation "Delete" signal, and using that to perform a second restart if necessary.

Specifically, we can:

• Ensure the resource delete is not unnecessarily invoked by having a resource ID that reflects the dependent parameters (eg. a hash of all their names)
• Have the resource delete only restart after dependent parameters have been deleted (only restart when required) by getting it to introspect it's own stack and wait for any SSM Parameter Resources that are being deleted.

h/t @v-do for finding the bug

Support older Python versions

Versions required:

• Python 2.7
• Python 3 up to v3.5

Work (I think):

• Setup test runs with those versions
• Use six where necessary

Note that this is likely to be bigger than it looks, because flying-circus is also Python v3.6+ (for the same reasons as us, mainly because f strings and type annotations are appealing)

Use multiple CLI command chaining

Use click's command chaining to string together multiple semi-independent transformations.

Installation fails due to conflicting pyyaml version

Hi, users are unable to run ssmash due to dependency conflict with pyyaml package. As shown in the following full dependency graph of ssmash, ssmash requires pyyaml >=5.1,<6，while flying-circus requires <5.2.0,>=5.1.1.

According to pip’s “first found wins” installation strategy, pyyaml 5.2 is the actually installed version. However, pyyaml 5.2 does not satisfy PyYAML<5.2.0,>=5.1.1.

Dependency tree-----------

ssmash - 2.0.1
| +- click(install version:7.0 version range:>=7.0,<8)
| +- flying-circus(install version:0.7.2 version range:>=0.7,<0.8)
| | +- attrs(install version:19.3.0 version range:>=18.2.0)
| | +- inflection(install version:0.3.1 version range:<0.4,>=0.3.1)
| | +- pyyaml(install version:3.13 version range:<5.2.0,>=5.1.1)
| +- inflection(install version:0.3.1 version range:==0.3.1)
| +- pyyaml(install version:5.2 version range:>=5.1,<6)

Thanks for your help.
Best,
Neolith

As a developer I am using best practices for python project management

• Dependency management: Dependency ranges, freeze transitive dependencies, etc.
• Config file: Do we use pyproject.toml or setup.cfg or ...
• Inject dependencies into various places in setup.py

Move AWS-specific hypothesis strategies into an external library

Move AWS-specific hypothesis strategies into an external library

As a developer my code gets built automatically

• pre-merge build
• post-merge release
• style checks
• tests

Put invalidation config into YAML file

As a user I can specify invalidation parameters inside the YAML config file, so that I can invalidate services for subsections of the parameters

Provide CLI options to upload to AWS

There is a non-trivial amount of shell scripting fuss involved in calling ssmash to create a cloud formation template, then uploading that template to cloudformation (with the right parameters), then potentially restarting your ECS service or whatever else.

Add some CLI options to make this process built-in to ssmash. It's within the scope of what we want to do ("make it easy to manage application configuration in SSM Parameter Store"), and would deliver reasonable value.

We don't really want to re-implement the create-or-update semantics of the AWS CLI deploy command, but perhaps see https://github.com/aws/aws-cli/blob/develop/awscli/customizations/cloudformation/deploy.py for a minimal amount of skeleton code we could copy

As a user I can convert an app config specification into a CloudFormation template

As a user I can convert a simple app config specification in YAML, into an AWS CloudFormation template that creates SSM parameters

Clean up unwanted cookiecutter output