foremast / foremast

Spinnaker Pipeline/Infrastructure Configuration and Templating Tool - Pipelines as Code.

Spinnaker Foremast

Foremast is a Spinnaker pipeline and infrastructure configuration and templating tool. Just create a couple JSON configuration files and then manually creating Spinnaker pipelines becomes a thing of the past.

Why Foremast?

• No manual creation of pipelines in the Spinnaker UI
• Reproducible and versioned Spinnaker pipelines
• Standardized pipelines with flexibilty for application specific needs

With Foremast, Developers create a couple simple JSON configs per application. These configs provide details on the pipeline and infrastructure specific to the application's needs. Foremast takes those configs, renders some Jinja2 templates, and then acts as a client for the Spinnaker Gate API. Foremast comes with generic templates for creating a simple pipeline but it can also point to external templates for custom pipelines that fit any workflow.

Foremast Features

• Dynamically generate Spinnaker pipelines based on JSON configs
• Customizable pipelines through external Jinja2 Templates, see Foremast templates for examples
• Dynamically generate AWS infrastructure based on pipeline configs
• Set up resources not defined in Spinnaker, such as S3 buckets and IAM roles
• Support for AWS Lambda pipelines

Getting Started

Take a look at quick start guide for a quick introduction on how to use Foremast.

We also have a blog post to help you get started: Automate Spinnaker Pipeline Creation

Documentation

All the documentation can be viewed on Read the Docs. You can find all configuration options, code information, and better examples there.

Development

See the contribution guide for information on code style, contributing, and testing.

Getting Help

For questions, support, or friendly conversation you can find us on Gitter.

More Details

Installing

Installing the package will provide CLI commands for convenience.

virtualenv -p python3 venv
source venv/bin/activate
pip install foremast

Entry Points

Foremast has a few easy to use CLI endpoints.

• foremast-pipeline - Creates an application and pipeline Spinnaker
• foremast-infrastructure - Sets up AWS infrastructure like s3, iam, elb, and security groups
• foremast-pipeline-onetime - Generates a pipeline for deploying to one specific account
• foremast-scaling-policy - Creates and attaches a scaling policy to an application server group.
• foremast-pipeline-rebuild - rebuild pipelines after changes have been made

You can run any of these entries points from the command line. They rely on environment variables and are ideal for running in a Jenkins job

PROJECT=forrest GIT_REPO=core RUNWAY_DIR=path/to/pipeline_configs foremast-pipeline

Foremast Configuration

A file at {pwd}/.foremast/foremast.cfg, ~/.foremast/foremast.cfg, or /etc/foremast/foremast.cfg needs to exist in order to run foremast.

[base]
domain = example.com
envs = dev,stage,prod
regions = us-east-1
gate_api_url = http://gate.example.com:8084

Runway Configuration Files

To begin using Foremast, you must have a few JSON configuration files defined for each application

pipeline.json

This file will be needed for each application. Foremast has a lot of defaults in place for pipeline.json, take a look at the pipeline.json docs for all options.

Minimum

{
    "deployment": "spinnaker"
}

Example Deployment Environments Override

Custom deployment environment order and selection can be provided in the env key. When missing, the default provided is {"env": ["stage", "prod"]}. Here, the order matters and Pipeline will be generated in the given order.

{
    "deployment": "spinnaker",
    "env": [
        "prod"
    ]
}

application-master-{env}.json

Each deployment environment specified in the pipeline.json file will need an accompanying application-master-{env}.json file in the same directory.

The `application-master-{env} files have a lot of exposed values with sane defaults. Please take a look at the application.json docs for all options.

application-master-{env}.json example

{
  "security_group": {
    "description": "something useful",
    "elb_extras": ["sg_offices"],
    "ingress": {
    },
    "egress": "0.0.0.0/0"
  },
  "app": {
    "instance_type": "t2.small",
    "app_description": "Edge Forrest Demo application",
    "instance_profile": "forrest_edge_profile"
  },
  "elb": {
    "subnet_purpose": "internal",
    "target": "TCP:8080",
    "ports": [
      {"loadbalancer": "HTTP:80", "instance": "HTTP:8080"}
    ]
  },
  "asg": {
    "subnet_purpose": "internal",
    "min_inst": 1,
    "max_inst": 1,
    "scaling_policy": {
        "metric": "CPUUtilization",
        "threshold": 90,
        "period_minutes": 10,
        "statistic": "Average"
        }
  },
  "regions": ["us-east-1"],
  "dns" : {
    "ttl": 120
    }
}

foremast's Issues

Type "lambda" should auto-add service

When {"type": "lambda"} in pipeline.json, "services" should have {"lambda": true} added automatically.

Refactor renumerate_stages

https://github.com/gogoair/foremast/blob/master/src/foremast/pipeline/renumerate_stages.py

This can be refactored to be more agnostic to stage names. Right now it is kind of specific to the Gogo workflow but there should be a better way.

templates: ElasticSearch IAM statement

We need an ElasticSearch template in src/foremast/templates/infrastructure/iam/. Use src/foremast/templates/infrastructure/iam/dynamodb.json.j2 as a base example. Try to restrict access to creating and deleting Resources. Remove the for loop if resource targeting is not needed.

Example AWS provided AmazonESFullAccess Policy.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": [
      "es:*"
    ],
    "Effect": "Allow",
    "Resource": "*"
  }]
}

We expect vpc to be named 'vpc'.

We expect/assume that the vpc named for spinnaker is 'vpc'.
https://github.com/gogoair/foremast/blob/master/src/foremast/utils/vpc.py#L51

This may or may not be correct. In either case, we should better document our expectations.

stage-judgement-* template files are copy-pasta except for one field

These templates:

https://github.com/gogoair/foremast/blob/master/src/foremast/templates/pipeline/stage-judgement-nonprod.json.j2
https://github.com/gogoair/foremast/blob/master/src/foremast/templates/pipeline/stage-judgement-promote-s3-nonprod.json.j2
https://github.com/gogoair/foremast/blob/master/src/foremast/templates/pipeline/stage-judgement-promote-s3-prod.json.j2

Are entirely the same except for the instructions field (and the instructions are specific to Gogo). It looks like there is a need to have this customized per-environment and per type of pipeline (to override it on s3 to non-prod, and override it on s3 to prod)

datapipeline: Interpolate Subnets similar to EC2 Pipelines

Today, users only need to say internal or external to deploy to specific subnets based on metadata tags. We should have the same functionality in our datapipeline feature where users can specify internal and external and we figure out the subnets for them.

Today, they need to maintain a static list of subnets which is annoying and less than ideal (especially since that information is sometime not always shared to non Admins)

Specify application name

It would be extremely helpful if there was a way to specify the application name that gets created.

Currently it takes the name by combining the repository name and the group name from git. That means that the repo name and group name combined must be shorter than 24 characters and must only contain alphanumeric characters. This leads to repos with names that are ambiguous and hard to read.

It seems like a pretty bad flow to have your pipelining tool dictate the names of your repo. If overwriting others' apps is a concern then the group name could be added to the configured name to ensure that only groups can overwrite their apps.

Better error when no env/region for infrastructure prepare

These fails on eureka but it should fail if no env or region is provided and be clear why it is failing. I spent 30 minutes trying to figure though :p

Reenable linting once prospector is fixed

landscapeio/prospector#205

Once this PR is accepted we can reenable linting in our tests.

Retain Running Pipeline Configuration Post Update

Example Flow:
DEV --> STAGE --> PROD
/health /health /health

Found an issue where a developer may have a running pipeline deployed to STAGE. If I come in and update a pipeline configuration to change say the health endpoint to /status, it will potentially break existing pipelines running that may have code that only listens at /health.

First pipeline:
DEV --> STAGE --> PROD
/health /health /health

Second Pipeline:
DEV --> STAGE
/health /health

Third Pipeline:
DEV -->
/status

Second Pipeline (promote PROD):
DEV --> STAGE --> PROD !!!!!
/health /health /status !!!!!

templates: Kinesis Firehose IAM statement

We need a Kinesis Firehose template in src/foremast/templates/infrastructure/iam/. Use src/foremast/templates/infrastructure/iam/dynamodb.json.j2 as a base example to support a list of Delivery Streams.

Example AWS provided AmazonKinesisFirehoseFullAccess Policy.

{
  "Version": "2012-10-17",
  "Statement": [
  {
    "Action": [
    "firehose:*"
  ],
  "Effect": "Allow",
  "Resource": "*"
  }]
}

Lambda creation/code update should create ALIAS with ENV name

When creating Lambda function or updating lambda function we should alias $latest with ENV name so developers can easily fetch environment name and apply proper configs. That way they do not need to know account numbers or any other info.

Having lambda configs for all environments locally shortens the execution time (by not having to lookup external configs/services) and enables us to have single artifact for multiple ENVs for lambdas that need to have multiple different configs (for example ones that query Eureka for services).

This in turn needs to apply triggers to ENV Alias and not to $latest as it was the case which will leave current triggers defined on $latest for already deployed functions.

I can submit PR for this functionality if everyone agrees we need this kind of functionality.

foremast-infrastructure is not creating the s3 bucket

foremast-infrastructure is looking for existing s3 bucket instead of creating one defined in foremast.cfg -> [formats]
s3_bucket = bucket-{project}-{env}

Trace:

Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/foremast/s3/create_archaius.py", line 51, in init_properties
s3client.Object(archaius['bucket'], archaius_file).get()
File "/usr/local/lib/python3.6/site-packages/boto3/resources/factory.py", line 520, in do_action
response = action(self, *args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/boto3/resources/action.py", line 83, in call
response = getattr(parent.meta.client, operation_name)(**params)
File "/usr/local/lib/python3.6/site-packages/botocore/client.py", line 310, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/usr/local/lib/python3.6/site-packages/botocore/client.py", line 599, in _make_api_call
raise error_class(parsed_response, operation_name)
botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the GetObject operation: The specified bucket does not exist

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/bin/foremast-infrastructure", line 11, in
load_entry_point('foremast==3.19.4.dev1+ge1725fe', 'console_scripts', 'foremast-infrastructure')()
File "/usr/local/lib/python3.6/site-packages/foremast/runner.py", line 265, in prepare_infrastructure
runner.create_archaius()
File "/usr/local/lib/python3.6/site-packages/foremast/runner.py", line 132, in create_archaius
s3.init_properties(env=self.env, app=self.app)
File "/usr/local/lib/python3.6/site-packages/foremast/s3/create_archaius.py", line 56, in init_properties
s3client.Object(archaius['bucket'], archaius_file).put()
File "/usr/local/lib/python3.6/site-packages/boto3/resources/factory.py", line 520, in do_action
response = action(self, *args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/boto3/resources/action.py", line 83, in call
response = getattr(parent.meta.client, operation_name)(**params)
File "/usr/local/lib/python3.6/site-packages/botocore/client.py", line 310, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/usr/local/lib/python3.6/site-packages/botocore/client.py", line 599, in _make_api_call
raise error_class(parsed_response, operation_name)
botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the PutObject operation: The specified bucket does not exist

Add IAM requirements to documentation

The documentation should include the IAM requirements for foremast to run successfully.
​
Foremast touches Route 53, Lambda, S3, security groups, IAM, and other AWS services using boto3 outside of Spinnaker's API.

Plugin System

I've been really interested in setting up a plugin system which I believe will help the project tremendously.

The experimentation has been occurring in gogoair/foremast-plugin-experiment-1.

My suggestion is to reorganize the project, so its something like this:

└── src
    ├── foremast
    │   ├── common
    │   │   └── base.py
    │   ├── dns
    │   │   ├── aws.py
    │   │   ├── base.py
    │   │   └── gcp.py
    │   └── loadbalancer
    │       ├── aws.py
    │       ├── base.py
    │       └── gcp.py

The general idea is that we should be able to create resources for a specific 'provider', as such:

my_plugin = plugin_source.load_plugin('aws')
dns = my_plugin.Dns()
dns.create(**kwargs)

We'd still have to define specific plugin resource 'structure' but as a first pass, I'm suggesting:

• create()
• update()
• destroy()

How does this sound?

Add chaos support

add support so that chaos monkey can be configured in pipeline.json or app.json. Chaos monkey in spinnaker is configured in the application creation and the json looks like:

    "chaosMonkey": {
      "enabled": true,
      "meanTimeBetweenKillsInWorkDays": 1,
      "minTimeBetweenKillsInWorkDays": 1,
      "grouping": "cluster",
      "regionsAreIndependent": true,
      "exceptions": [
        {
          "account": "prod",
          "stack": "*",
          "region": "*",
          "detail": "*"
        },
        {
          "account": "prods",
          "region": "*",
          "stack": "*",
          "detail": "*"
        },
        {
          "account": "prodp",
          "region": "*",
          "stack": "*",
          "detail": "*"
        }
      ]
    },

This will need to be part of the payload on create-app stage.

Disable / Rework environment specific rules

In the following code below we are ensuring that the 'dev' environment has a few specific settings. Now, this applies to gogo but may not be a good idea for other users to have the same restrictions. Other foremast users probably have an account named 'dev' and are likely wondering why those asg min/max settings aren't adhered to.

The code in question is:
https://github.com/gogoair/foremast/blob/master/src/foremast/pipeline/construct_pipeline_block.py#L135-L143

We need to figure out a better way to allow this functionality or remove this stuff completely.

"lambda_triggers" should default to empty list

Unsafe access at https://github.com/gogoair/foremast/blob/master/src/foremast/awslambda/awslambdaevent.py#L41. Need to set a default for "lambda_triggers", most likely to {"lambda_triggers": []} for application-{env}.json files.

Lambda create_function() should retry

Due to the delay that happens when modifying a Role's Inline Policy, we need to retry a few times.

Docs for S3 deployments

Update/Remove Tested Versions in Docs

Looking through the docs, I noticed our tested versions is way out of date. I can see this as being a pain to update, we should either remove it or update it.

http://foremast.readthedocs.io/en/latest/infra_assumptions.html

Address TODO regarding region overrides in pipeline creation

https://github.com/gogoair/foremast/blob/master/src/foremast/pipeline/create_pipeline.py#L220

fix: Check `pyapi-gitlab` returns

Problem

Need to check pyapi-gitlab returns before trying to use the returned object. For request errors, False is returned instead of a more useful object.

There has been some work in pyapi-gitlab/pyapi-gitlab#238 to raise instead of return None. Switching to handle exceptions would make this a little easier.

Example:

Should check the return of .getbranch() before trying to access like a dict.

# src/foremast/configs/prepare_configs.py
    config_commit = file_lookup.server.getbranch(file_lookup.project_id, 'master')['commit']['id']

Add Load Balancer and time-based Stickiness for ELBs

Add better logging and docs around RUNWAY_DIR for jenkins jobs

It is not mentioned in the docs that you need RUNWAY_DIR for the prepare jenkins jobs if you are not using Gitlab. The error messages should be better and the docs should mention that it needs passed in.

Fail early if no region is provided

Right now if region is not passed in during running, you can get a lot of weird and random errors that are not clear.

Enable toggling of Jenkins trigger

There are some instances where users don't want pipelines to automatically trigger on Jenkins job completion and they prefer to run it manually.

Default lambda IAM role should include Cloudwatch

Attached lambda iam role is missing cloudwatch permissions which in turn doesn't allow for lambda logging which will create confusion with developers.

Documentation around MultiRegion Failover

We rolled out some basic functionality of creating multi region DNS with simple failover. It would be good to get some basic documentation around this before we commit too much ahead of ourselves!

Better documentation around custom templates

Enable DynamoDB Triggers to AWS Lambda Pipeline

Currently Foremast Lambda Pipeline supports: api-gateway, s3, sns, cloudwatch-event, cloudwatch-log

Enable DynamoDB triggers.E.g. when a record is inserted OR updated a lambda is triggered. This is achieved manually today e.g. devops need to manually add the trigger after deploying via pipeline

feat: Manual deployment type

Provide the ability to have foremast manage a predefined Pipeline using the raw JSON. This will be allow one to copy the JSON from the Pipeline, save it in a repository, and have foremast reconfigure the defined Pipeline with every run.

Query exectionEngine to support Orca 2.x

from Gitter Chat:

if you check one of your existing pipelines (that was built manually) you should see the executionEngine:v2, if not then yeah your Spinnaker version is a bit old. In any case we made the change in the src/foremast/templates/pipeline/pipeline_wrapper.json.j2 template.

Might be possible to query for that execution engine so that it does not need to be configured

   {% endif %}
   "application": "{{ data.app.appname }}",
+  "executionEngine": "v2",
   "triggers": [
     {% include "pipeline/trigger-jenkins.json.j2" %}
   ],```

Pipeline JSON Validator

We should have an intermediary step to check the JSON before posting to Spinnaker. On top of that, this enables people to develop plugins to modify pipeline JSON based on business logic (keeping foremast clean). For example, if environment is dev, update max ASG size from 3 to 1.

Raw code logic is something like:

config_check=True
config_checker=<something>

if config_check:
    json = check_json(config_checker, json_load)
post_spinnaker(json)

Cleans up things like #123

What should happen if you revert multi-region?

How should the DNS records be handled if your application goes back to Single Region after being Multi-Region?

Onetime pipeline appears to only work with single region

If a specific account has multiple regions defined, we randomly pick one, which may not be the region you were hoping for.

We should either:

1. Ask for which region in onetime
2. Create onetime pipelines for all regions

Add config validation before running anything

Right now the errors are pretty ambiguous if one of the runway config files is missing or has a json error. We could run a config validation stage before doing anything else. Make sure that all the needed applicaiton-master-*.json files exist, pipeline.json exists, and that the JSON is valid.

Add AWS S3 Glacier support

Also it would be great to be able to specify life cycle policies in pipeline config files.

Region Specific application json configs

The application json config should be region specific. You may want different settings in one region vs another

Problems with Spinnaker LDAP authentication

Thanks for a great tool. I'm having some issues running it against my Spinnaker instance because it requires a login via LDAP.

When I run "foremast validate all", I get the following output:

2017-11-14 20:34:25,295 [INFO] foremast.consts:find_config:131 - Loading static configuration file.
2017-11-14 20:34:25,295 [WARNING] foremast.consts:validate_key_values:67 - [base] missing key "git_url", using None.
2017-11-14 20:34:25,295 [WARNING] foremast.consts:validate_key_values:67 - [base] missing key "types", using 'ec2,lambda,s3,datapipeline'.
2017-11-14 20:34:25,296 [WARNING] foremast.consts:validate_key_values:67 - [base] missing key "ami_json_url", using None.
2017-11-14 20:34:25,296 [WARNING] foremast.consts:validate_key_values:67 - [base] missing key "default_securitygroup_rules", using ''.
2017-11-14 20:34:25,296 [WARNING] foremast.consts:validate_key_values:67 - [base] missing key "default_ec2_securitygroups", using ''.
2017-11-14 20:34:25,297 [WARNING] foremast.consts:validate_key_values:67 - [base] missing key "default_elb_securitygroups", using ''.
2017-11-14 20:34:25,297 [INFO] foremast.consts:validate_key_values:58 - Section missing from configurations: [credentials]
2017-11-14 20:34:25,297 [WARNING] foremast.consts:validate_key_values:67 - [credentials] missing key "gitlab_token", using None.
2017-11-14 20:34:25,298 [WARNING] foremast.consts:validate_key_values:67 - [credentials] missing key "slack_token", using None.
2017-11-14 20:34:25,298 [INFO] foremast.consts:validate_key_values:58 - Section missing from configurations: [task_timeouts]
2017-11-14 20:34:25,298 [WARNING] foremast.consts:validate_key_values:67 - [task_timeouts] missing key "default", using 120.
2017-11-14 20:34:25,298 [WARNING] foremast.consts:validate_key_values:67 - [task_timeouts] missing key "envs", using '{}'.
2017-11-14 20:34:25,298 [INFO] foremast.consts:validate_key_values:58 - Section missing from configurations: [whitelists]
2017-11-14 20:34:25,299 [WARNING] foremast.consts:validate_key_values:67 - [whitelists] missing key "asg_whitelist", using ''.
2017-11-14 20:34:25,299 [WARNING] foremast.consts:validate_key_values:67 - [base] missing key "gate_client_cert", using ''.
2017-11-14 20:34:25,299 [WARNING] foremast.consts:validate_key_values:67 - [base] missing key "gate_ca_bundle", using ''.
2017-11-14 20:34:25,299 [INFO] foremast.consts:validate_key_values:58 - Section missing from configurations: [links]
2017-11-14 20:34:25,300 [WARNING] foremast.consts:validate_key_values:67 - [links] missing key "default", using '{}'.
2017-11-14 20:34:25,558 [INFO] foremast.validate:validate_all:24 - Running all validate steps.
InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
Traceback (most recent call last):
  File "/usr/local/bin/foremast", line 11, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.6/site-packages/foremast/__main__.py", line 116, in main
    args.parsed.func(args)
  File "/usr/local/lib/python3.6/site-packages/foremast/validate.py", line 25, in validate_all
    validate_gate()
  File "/usr/local/lib/python3.6/site-packages/foremast/validate.py", line 13, in validate_gate
    credentials = get_env_credential()
  File "/usr/local/lib/python3.6/site-packages/foremast/utils/credentials.py", line 81, in get_env_credential
    credential = credential_response.json()
  File "/usr/local/lib/python3.6/site-packages/requests/models.py", line 892, in json
    return complexjson.loads(self.text, **kwargs)
  File "/usr/local/lib/python3.6/json/__init__.py", line 354, in loads
    return _default_decoder.decode(s)
  File "/usr/local/lib/python3.6/json/decoder.py", line 339, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/local/lib/python3.6/json/decoder.py", line 357, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

Upon closer inspection, this appears to be because the Gate URL is returning the login form as HTML, instead of the expected JSON:

(Pdb) l
352
353  	        """
354  	        try:
355  	            obj, end = self.scan_once(s, idx)
356  	        except StopIteration as err:
357  ->	            raise JSONDecodeError("Expecting value", s, err.value) from None
358  	        return obj, end
[EOF]
(Pdb) p s
'<html><head><title>Login Page</title></head><body onload=\'document.f.username.focus();\'>\n<h3>Login with Username and Password</h3><form name=\'f\' action=\'/login\' method=\'POST\'>\n<table>\n\t<tr><td>User:</td><td><input type=\'text\' name=\'username\' value=\'\'></td></tr>\n\t<tr><td>Password:</td><td><input type=\'password\' name=\'password\'/></td></tr>\n\t<tr><td colspan=\'2\'><input name="submit" type="submit" value="Login"/></td></tr>\n</table>\n</form></body></html>'

Any suggestions on how to make this work?

Multiplying a string

If someone has the following asg policy:

  "asg": {
    "subnet_purpose": "internal",
    "min_inst": 1,
    "max_inst": 5,
    "scaling_policy": {
      "metric": "NetworkIn",
      "threshold": "20000",
      "period_minutes": 1,
      "statistic": "Minimum"
    }
  },

You will see an error, like so:

packages/foremast/autoscaling_policy/create_policy.py", line 79, in prepare_policy_template
    'asg']['scaling_policy']['threshold'] * 0.5
TypeError: can't multiply sequence by non-int of type 'float'

The reason is that threshold is a string. If threshold is an int, this works.

We should ensure that thresholds are typecast to int(), as a possible fix for this issue.

bug: Example deploy template needs max ASGs for redblack deployments

redblack deployments will default to empty without the "maxRemainingAsgs" key defined, which turns into keeping an unlimited number of ASGs. We should provide the minimum number of 2 as a safe default.

create-scaling-policy should use module entry point

create-scaling-policy currently uses the runner endpoint to get executed. I think it should probably use the module (__main__.py) entry point instead of the runner.

This aligns that entry point like other create-* ones.

templates: ElastiCache IAM statement

We need an ElastiCache template in src/foremast/templates/infrastructure/iam/. Use src/foremast/templates/infrastructure/iam/dynamodb.json.j2 as a base example. Try to restrict access to creating and deleting Resources. Remove the for loop if resource targeting is not needed.

Example AWS provided AmazonElastiCacheFullAccess Policy.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": "elasticache:*",
      "Effect": "Allow",
      "Resource": "*"
    }
  ]
}

Update foremast.cfg to include x509 example

We need to update the docs for foremast.cfg to include examples on using x509 when authenticating against gate.

Remove /13 security restrictions when creating security groups

I'm proposing we remove the following lines of code:
https://github.com/gogoair/foremast/blob/master/src/foremast/securitygroup/create_securitygroup.py#L93-L95

We should not expect external users to have the same restriction we have in our environments.

This will allows users to specify any IP cidr for their security rules.

Decrease scaling policies are not attaching theshold in AWS

Looks like when creating the decrease scaling policy, odd numbers do not work because it expects an int but gets a float.

Like if you increase at 55% CPU, it should decrease at 27.5% but this does not work.

Clean up removed Lambda triggers

If you remove a lambda trigger from your pipeline configs, it should be removed from Lambda. Currently, if removed or name is changed it does not clean anything up and can cause duplicate or abandoned triggers

feat: Add support for setting Public IP

Add an option to disable or enable the Public IP allocation. Will need to figure out which values the "associatePublicIpAddress" takes in the stage-deploy.json.j2.

{
          "associatePublicIpAddress": null
}

Refactor create_pipeline_* logic

There is a lot of duplicated code between all the pipeline types. This can be refactored to be a lot more simple and clear