onicagroup / runway

Simplify infrastructure/app testing/deployment

Runway

Runway is a lightweight integration app designed to ease management of infrastructure tools.

Its main goals are to encourage GitOps best-practices, avoid convoluted Makefiles/scripts (enabling identical deployments from a workstation or CI job), and enable developers/admins to use the best tool for any given job.

Features

• Centralized environment-specific configuration
• Automatic environment identification from git branches
• Automatic linting/verification
• Support of IAM roles to assume for each deployment
• Terraform backend/workspace config management w/per-environment tfvars
• Automatic kubectl/terraform version management per-environment

Supported Deployment Tools

• AWS CDK
• Kubectl
• Serverless Framework
• CFNgin (CloudFormation)
• Static websites (build & deploy to S3+CloudFront)
• Terraform

Example

A typical Runway configuration is unobtrusive -- it just lists the deployment order and locations (regions).

deployments:
  - modules:
      - resources.tf  # terraform resources
      - backend.sls  # serverless lambda functions
      - frontend  # static web site
    environments:  # Environments
      dev: "123456789012"  # AWS development Account ID
      prod: "234567890123"  # AWS production Account ID
    regions:
      - us-east-1

The example above contains enough information for Runway to deploy all resources, lambda functions and a static website backed by S3 and Cloudfront in either dev or prod environments

Install

Runway is available via any of the following installation methods. Use whatever works best for your project/team (it's the same application no matter how you obtain it).

HTTPS Download (e.g cURL)

Use one of the endpoints below to download a single-binary executable version of Runway based on your operating system.

Operating System	Endpoint
Linux	https://oni.ca/runway/latest/linux
macOS	https://oni.ca/runway/latest/osx
Windows	https://oni.ca/runway/latest/windows

$ curl -L oni.ca/runway/latest/osx -o runway
$ chmod +x runway
$ ./runway new

Suggested use: CloudFormation or Terraform projects

npm

$ npm i -D @onica/runway
$ npx runway new

Suggested use: Serverless or AWS CDK projects

pip (or poetry, etc)

$ pip install runway
$ runway new
# OR
$ poetry add --dev runway
$ poetry run runway new

Suggested use: Python projects

Documentation

See the doc site for full documentation.

Quickstart documentation, including CloudFormation templates and walkthrough can be found here

runway's Issues

runway (0.8.0rc2) assumerole in config and logged as being used, but gets denied as original role when attempting to do a CreateStacks

I have an instance with a role, that can pretty much do nothing. We have a second role that allows for cloudformation deployments only that can be assumed by the original role. This works when I use the aws cli to manually assume the role before a runway takeoff, but fails when I instead rely on runway to assume the role for me.

Correct role in runway.yml: arn:aws:iam::########:role/assumerol-AssumeRole-1GXCX51B43A27
Role instance was launched with: arn:aws:sts::#####:assumed-role/instance17-ServerRole-OYAM3DY5SD4Q

My runway.yml:

deployments:
  - modules:
      - fargate.cfn
    regions:
      - us-east-1
    assume-role:
      arn: arn:aws:iam::########:role/assumerol-AssumeRole-1GXCX51B43A27
ignore_git_branch: true

Log of my Jenkins log with failure:

13:54:29 --------------------------------------------------------------------------------
13:54:29 executing runway takeoff...
13:54:29 --------------------------------------------------------------------------------
13:54:29 INFO:botocore.vendored.requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): 169.254.169.254
13:54:29 INFO:botocore.vendored.requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): 169.254.169.254
13:54:29 INFO:runway:Assuming role arn:aws:iam::########:role/assumerol-AssumeRole-1GXCX51B43A27...
13:54:29 INFO:botocore.vendored.requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): sts.amazonaws.com
13:54:29 INFO:runway:Skipping environment lookup from current git branch ("ignore_git_branch" is set to true in the runway config)
13:54:29 INFO:runway:Deriving environment name from directory /app/jenkins/workspace/Dashboard_Promote/dev...
13:54:29 INFO:runway:Deploying to dev environment in region us-east-1...
13:54:29 INFO:runway:Running stacker build on fargate.yaml
13:54:29 INFO:runway.embedded.stacker.commands.stacker:Using interactive AWS provider mode.
13:54:29 INFO:runway.embedded.stacker.plan:Plan Status:
13:54:29 INFO:runway.embedded.stacker.plan:	dev-dashboard-fargate: pending

13:54:30 Traceback (most recent call last):
13:54:30   File "/app/jenkins/workspace/Dashboard_Promote/venv/bin/runway", line 11, in <module>
13:54:30     sys.exit(main())
13:54:30   File "/app/jenkins/workspace/Dashboard_Promote/venv/lib/python2.7/site-packages/runway/cli.py", line 56, in main
13:54:30     command.execute()
13:54:30   File "/app/jenkins/workspace/Dashboard_Promote/venv/lib/python2.7/site-packages/runway/commands/takeoff.py", line 10, in execute
13:54:30     self.deploy()
13:54:30   File "/app/jenkins/workspace/Dashboard_Promote/venv/lib/python2.7/site-packages/runway/commands/env.py", line 84, in deploy
13:54:30     self.run(deployments=deployments, command='deploy')
13:54:30   File "/app/jenkins/workspace/Dashboard_Promote/venv/lib/python2.7/site-packages/runway/commands/env.py", line 68, in run
13:54:30     command)()
13:54:30   File "/app/jenkins/workspace/Dashboard_Promote/venv/lib/python2.7/site-packages/runway/commands/module.py", line 431, in deploy
13:54:30     command='build'
13:54:30   File "/app/jenkins/workspace/Dashboard_Promote/venv/lib/python2.7/site-packages/runway/commands/module.py", line 346, in run_stacker
13:54:30     args.run(args)
13:54:30   File "/app/jenkins/workspace/Dashboard_Promote/venv/lib/python2.7/site-packages/runway/embedded/stacker/commands/stacker/build.py", line 46, in run
13:54:30     dump=options.dump)
13:54:30   File "/app/jenkins/workspace/Dashboard_Promote/venv/lib/python2.7/site-packages/runway/embedded/stacker/actions/base.py", line 131, in execute
13:54:30     self.run(*args, **kwargs)
13:54:30   File "/app/jenkins/workspace/Dashboard_Promote/venv/lib/python2.7/site-packages/runway/embedded/stacker/actions/build.py", line 366, in run
13:54:30     plan.execute()
13:54:30   File "/app/jenkins/workspace/Dashboard_Promote/venv/lib/python2.7/site-packages/runway/embedded/stacker/plan.py", line 292, in execute
13:54:30     if not self._single_run():
13:54:30   File "/app/jenkins/workspace/Dashboard_Promote/venv/lib/python2.7/site-packages/runway/embedded/stacker/plan.py", line 252, in _single_run
13:54:30     status = step.run()
13:54:30   File "/app/jenkins/workspace/Dashboard_Promote/venv/lib/python2.7/site-packages/runway/embedded/stacker/plan.py", line 81, in run
13:54:30     return self._run_func(self.stack, status=self.status)
13:54:30   File "/app/jenkins/workspace/Dashboard_Promote/venv/lib/python2.7/site-packages/runway/embedded/stacker/actions/build.py", line 294, in _launch_stack
13:54:30     force_change_set=force_change_set
13:54:30   File "/app/jenkins/workspace/Dashboard_Promote/venv/lib/python2.7/site-packages/runway/embedded/stacker/providers/aws/default.", line 799, in update_stack
13:54:30     **kwargs)
13:54:30   File "/app/jenkins/workspace/Dashboard_Promote/venv/lib/python2.7/site-packages/runway/embedded/stacker/providers/aws/default.py", line 819, in interactive_update_stack
13:54:30     'UPDATE', service_role=self.service_role, **kwargs
13:54:30   File "/app/jenkins/workspace/Dashboard_Promote/venv/lib/python2.7/site-packages/runway/embedded/stacker/providers/aws/default.py", line 316, in create_change_set
13:54:30     kwargs=args
13:54:30   File "/app/jenkins/workspace/Dashboard_Promote/venv/lib/python2.7/site-packages/runway/embedded/stacker/providers/aws/default.py", line 81, in retry_on_throttling
13:54:30     retry_checker=_throttling_checker)
13:54:30   File "/app/jenkins/workspace/Dashboard_Promote/venv/lib/python2.7/site-packages/runway/embedded/stacker/util.py", line 76, in retry_with_backoff
13:54:30     return function(*args, **kwargs)
13:54:30   File "/app/jenkins/workspace/Dashboard_Promote/venv/lib/python2.7/site-packages/botocore/client.py", line 314, in _api_call
13:54:30     return self._make_api_call(operation_name, kwargs)
13:54:30   File "/app/jenkins/workspace/_Dashboard_Promote/venv/lib/python2.7/site-packages/botocore/client.py", line 612, in _make_api_call
13:54:30     raise error_class(parsed_response, operation_name)
13:54:30 botocore.exceptions.ClientError: An error occurred (AccessDenied) when calling the CreateChangeSet operation: User: arn:aws:sts::#####:assumed-role/instance17-ServerRole-OYAM3DY5SD4Q/i-##### is not authorized to perform: cloudformation:CreateChangeSet on resource: arn:aws:cloudformation:us-east-1:#####:stack/dev-dashboard-fargate/*
13:54:30 Build step 'Execute shell' marked build as failure
13:54:30 Finished: FAILURE

[Request] Add a path parameter

When using runway via a Docker container, it can get tricky to have Runway configurations in a directory different from your source code. It would be nice to have runway accept a path parameter that it would use to search for the runway.yml file.

Then the file paths within the runway.yml would still have to be relative to that file.

[REQUEST] drop python 2 support

• ETA December 31, 2020
• must be accompanied by a major version bump

Specify menu choices via the command-line

Background
When you execute Runway without the CI environment variable then you are presented a choice of which deployments and modules should be deployed. For example, you might see this:

$ runway deploy
...

Configured deployments:
1: base.cfn, databases.tf (us-east-1)
2: api.sls, frontend.sls (us-east-1, us-west-1, us-west-2)

Enter number of deployment to run (or "all"): 2

Configured modules in deployment:
1: api.sls ({...environment values...})
2: frontend.sls ({...environment values...})

Enter number of module to run (or "all"): 1

Those particular choices would result in only the frontend.sls module being deployed.

This is intended to be a time-saver for developers who are working on a particular module; it saves them from waiting for Runway to go through all of the other modules to decide they haven't changed. It's not intended to be used to deploy to "real" environments.

As nice as that is, choosing menu options can be error-prone, and it's easy to forget about, leaving you wondering why a long deploy didn't run while you went for coffee.

Proposal
Allow the two menu selections to be provided on the command-line:

$ runway deploy  --deployment-index 2 --module-index 2

The values provided should mimic exactly the selections from the menus; so for instance this would result in deploying everything:

$ runway deploy  --deployment-index all

And this would deploy the second deployment entirely:

$ runway deploy  --deployment-index 2 --module-index all

[BUG] report

Describe the bug
parallel_regions not kicking off modules in the same deployment in parallel

tested on runway 1.4.0 and 1.4.4

To Reproduce

1. clone https://github.com/animaxcg/runway-stacker-test
2. change the bucket to the desired deployment back
3. run start.sh
4. see that it kicks off regions simultaneously but not modules

Expected behavior
4. see that it kicks off regions simultaneously and modules

[REQUEST] feature

Is your feature request related to a problem? Please describe.
I want to be able to have a central directory for my CFNgin configs instead of having them in the module
Describe the solution you'd like
in the runway/runway/cfngin.py there should be a class attribute for config_directory that is passed down from runway.yml or runway.variables.yml or as a command line param (probably the form and not the latter 2)

Describe alternatives you've considered
each module can point to their own env directory

Additional context
It is frustrating when you have multiple modules that share much of the same configuration for stacker runs.

Runway issue while updating the Cloudformation Stack

We are facing Runway issue while updating the Cloudformation Stack after changing the value in the env file. While updating the stack with the new value, the job fails with the below error.
However, rerunning the job passes after deleting the stack. Please find the job details below

Error:
Changes

• Modify ParameterStore (AWS::SSM::Parameter)
  [2018-12-04T15:19:27] EOF when reading a line
  Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/runway/embedded/stacker/plan.py", line 89, in _run_once
  status = self.fn(self.stack, status=self.status)
  File "/usr/lib/python2.7/site-packages/runway/embedded/stacker/actions/build.py", line 311, in _launch_stack
  stack_policy=stack_policy,
  File "/usr/lib/python2.7/site-packages/runway/embedded/stacker/providers/aws/default.py", line 798, in update_stack
  stack_policy=stack_policy, **kwargs)
  File "/usr/lib/python2.7/site-packages/runway/embedded/stacker/providers/aws/default.py", line 838, in interactive_update_stack
  include_verbose=True,
  File "/usr/lib/python2.7/site-packages/runway/embedded/stacker/providers/aws/default.py", line 148, in ask_for_approval
  '/'.join(approval_options)))
  File "/usr/lib/python2.7/site-packages/runway/embedded/stacker/ui.py", line 48, in ask
  return get_raw_input(message)
  File "/usr/lib/python2.7/site-packages/runway/embedded/stacker/ui.py", line 11, in get_raw_input
  return raw_input(message)
  EOFError: EOF when reading a line
  [2018-12-04T15:19:27] aws-dev-parameter-al-img changes:
  Parameters Modified: KeyDescription, KeyValue

Changes:

• Modify ParameterStore (AWS::SSM::Parameter)
  [2018-12-04T15:19:27] EOF when reading a line

Please help us to fix this.

[Request] Add support for S3 default encryption on runway's S3 bucket

We run in an environment where all buckets require default encryption. Any plans on supporting AES and KMS key support for runway's S3 bucket? I'm not yet familiar with code base but can start taking a look as time permits.

[BUG] Lookup defaults cannot contain certain characters

Describe the bug
A default string value for a lookup that contains an = will cause the lookup resolution to fail

To Reproduce
Steps to reproduce the behavior:

1. Add a lookup with a default value containing a =.
2. Run runway plan or runway deploy
3. See error

Expected behavior
An escape syntax should be provided, or the lookup interpolation should be changed

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

• OS: Mac OS
• Version: 10.15.4

Additional context
An example of a runway.yml file with the issue is:

deployments:
  - modules:
      - path: infrastructure
        ...
        parameters:
          broken_var: ${var var_name.${env DEPLOY_ENVIRONMENT}::default=long_value_with_equal_sign_at_end=}

The output for this error is below:

runway.cfngin.exceptions.FailedVariableLookup: Couldn't resolve lookup in variable `infrastructure.parameters`, lookup: ${Lookup<Literal<'var'> Concatenation[Literal<'var_name.'>, Lookup<aoggz (Literal<'env'> Concatenation[Literal<'DEPLOY_ENVIRONMENT'>])>, Literal<'::default=long_value_with_equal_sign_at_end='>]>}: (<type 'exceptions.ValueError'>) too many values to unpack

I was investigating migrating to using the new parameters syntax over just using *-scoped env_vars as we had done in the past (with environment-specific env_vars that supercede the *-scoped vars). We have been using runway since 2017, so we have some old configuration files that we'd like to update. The wildcarded env_vars works, it's just not the greatest solution; the behavior of the configuration can be pretty obtuse to fresh eyes.

Edit:
I did forget to mention that I also tried to nest the default value back into the runway.variables.yml file:

${var var_name.${env DEPLOY_ENVIRONMENT}::default=${var var_name.default}}

This lead to a similar error:

runway.cfngin.exceptions.FailedVariableLookup: Couldn't resolve lookup in variable `infrastructure.parameters`, lookup: ${Lookup<Literal<'var'> Concatenation[Literal<'var_name.'>, Lookup<aoggz (Literal<'env'> Concatenation[Literal<'DEPLOY_ENVIRONMENT'>])>, Literal<'::default='>, Lookup<long_value_with_equal_sign_at_end= (Literal<'var'> Concatenation[Literal<'var_name.default'>])>, Literal<''>]>}: (<type 'exceptions.ValueError'>) too many values to unpack

gen-sample templates should use new staticsite options

cd runway/templates && grep -r "class_path: runway.module.staticsite.StaticSite" should return no results.

Minor: Fails to catch stacker error (MissingVariable)

Runway fails to catch an error thrown by stacker. I created a module, and forgot to include a variable (it was actually a CaSE SenSitVe issue), this caused runway to blow up, instead of repeating back to me the real error.

I'm nitpicking, but it would be nice if runway just showed the exact stack trace from stacker with it's own caught exception, instead of throwing a subprocess.CalledProcessError with a stack trace that is not helpful. (Believe it or not, the stacker exception scrolled off my screen)

INFO:runway:Running stacker build on application_definition.yaml
[2018-07-23T22:06:58] Using default AWS provider mode
[2018-07-23T22:06:59] sns-topics: skipped (nochange)
INFO:runway:Deploying to common environment in region us-east-1...
INFO:runway:Running stacker build on application_definition.yaml
[2018-07-23T22:06:59] Using default AWS provider mode
[2018-07-23T22:06:59] Variable "EnvironmentName" in blueprint "additional-subnets-1" is missing
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/stacker/plan.py", line 89, in _run_once
    status = self.fn(self.stack, status=self.status)
  File "/usr/local/lib/python2.7/dist-packages/stacker/actions/build.py", line 279, in _launch_stack
    stack.resolve(self.context, self.provider)
  File "/usr/local/lib/python2.7/dist-packages/stacker/stack.py", line 187, in resolve
    self.blueprint.resolve_variables(self.variables)
  File "/usr/local/lib/python2.7/dist-packages/stacker/blueprints/raw.py", line 139, in resolve_variables
    self.name
  File "/usr/local/lib/python2.7/dist-packages/stacker/blueprints/raw.py", line 63, in resolve_variable
    raise MissingVariable(blueprint_name, var_name)
MissingVariable: Variable "EnvironmentName" in blueprint "additional-subnets-1" is missing
[2018-07-23T22:06:59] additional-subnets-1: failed (Variable "EnvironmentName" in blueprint "additional-subnets-1" is missing)
[2018-07-23T22:06:59] The following steps failed: additional-subnets-1
Traceback (most recent call last):
  File "/usr/local/bin/runway", line 11, in <module>
    sys.exit(main())
  File "/usr/local/lib/python2.7/dist-packages/runway/cli.py", line 58, in main
    command.execute()
  File "/usr/local/lib/python2.7/dist-packages/runway/commands/deploy.py", line 10, in execute
    self.deploy()
  File "/usr/local/lib/python2.7/dist-packages/runway/commands/env.py", line 115, in deploy
    self.run(deployments=deployments, command='deploy')
  File "/usr/local/lib/python2.7/dist-packages/runway/commands/env.py", line 98, in run
    command)()
  File "/usr/local/lib/python2.7/dist-packages/runway/commands/module.py", line 525, in deploy
    command='build'
  File "/usr/local/lib/python2.7/dist-packages/runway/commands/module.py", line 433, in run_stacker
    env=self.env_vars
  File "/usr/lib/python2.7/subprocess.py", line 190, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/usr/bin/python', '-c', "import sys;sys.argv = ['stacker'] + ['build', '--region=us-east-1', '--recreate-failed', 'common-us-east-1.env', 'application_definition.yaml'];from stacker.logger import setup_logging;from stacker.commands import Stacker;stacker = Stacker(setup_logging=setup_logging);args = stacker.parse_args(['build', '--region=us-east-1', '--recreate-failed', 'common-us-east-1.env', 'application_definition.yaml']);stacker.configure(args);args.run(args)"]' returned non-zero exit status 1
Makefile:10: recipe for target 'deploy' failed
make: *** [deploy] Error 1
ERROR: Job failed: exit code 1

[TODO] remove stacker "shim"

• ETA December 31, 2020
• must accompany a major version bump

Documentation Error for Static Site Example

The example in the README shows the use of staticsite_cert_arn to presumably define a custom CloudFront cert using ACM. Configuration of this properly results in the CloudFront distribution using the default AWS-provided cert for CloudFront instead. The documentation in docs/staticsite.md seems accurate.

In the actual code for module/staticsite.py it looks as though it is expecting staticsite_acmcert_arn and not staticsite_cert_arn.

For example running grep -rE 'staticsite_(acm)?cert_arn' * yields:

README.md:            staticsite_cert_arn: arn:aws:acm:us-east-1:123456789012:certificate/...
README.rst:                staticsite_cert_arn: arn:aws:acm:us-east-1:123456789012:certificate/...
docs/staticsite.md:            staticsite_acmcert_arn: arn:aws:acm:us-east-1:123456789012:certificate/...
runway/module/staticsite.py:            site_stack_variables['AcmCertificateArn'] = '${default staticsite_acmcert_arn::undefined}'  # noqa pylint: disable=line-too-long

[Request] Option to reference Terraform backends from SSM parameters

Similarly to how runway can be configured to use CloudFormation stack outputs via the terraform_backend_cfn_outputs property, it would be useful to be able to tell runway to look at SSM parameters for the values of the Terraform backend configuration.

[Request] Have all linters and code checkers run when testing before exiting

Usually I find it speeds up testing if I can get all reported errors then I can fix them all in one go.

So instead of running flake8 and raising if it's not clean then running the next linter and so on, I'd like to see it where all of them are run and then exits non 0 if any of them failed.

incorrect url in quickstart

the link in
https://github.com/onicagroup/runway/blob/master/quickstarts/runway/README.md
for cloudformation should link to

https://github.com/onicagroup/runway/blob/master/quickstarts/runway/runway-quickstart.yml

not

https://github.com/onicagroup/runway/blob/master/quickstarts/runway-quickstart.yml

[BUG] Custom AWS_PROFILE, avoid default profile

Describe the bug
Not able to define AWS_PROFILE environment variable properly. I'm trying to set this variable to a value other than default and it's not working.

To Reproduce
Steps to reproduce the behavior:

1. Add AWS_PROFILE = custom_profile to your runway.yml
2. Run runway deploy

Expected behavior
AWS_PROFILE = custom_profile

Desktop (please complete the following information):

• OS: macOS Mojave
• Version 10.14.5
• Runway version 1.7.2
• Python version 3.7.7

Additional context
My profile is defined both in ~/.aws/credentials and in ~/.aws/config
I'm not sure if this is your issue or something related to boto3.
See also: https://stackoverflow.com/questions/33297172/boto3-error-botocore-exceptions-nocredentialserror-unable-to-locate-credential

Deployments should be programmatically selectable

Currently deployments can be interactively selected, or selection can be skipped entirely (via the CI environment variable).

This makes it impossible to do selective deployments/teardowns on a CI system (without resorting to brittle workarounds like echoing numbers into the runway's stdin).

An alternative to this should be provided. One option would be to allow optional tags to be specified on modules, and add a command line option to restrict operation to matching modules.

[REQUEST] support an AWS SSO profile as the initial credential source

Is your feature request related to a problem? Please describe.

When using AWS SSO profiles configured using awscli>=2.0.0 and selected with export AWS_PROFILE=<name>, the credentials cannot be loaded.

This means that in order to use Runway with an AWS SSO profile, the credentials would need to be collected and exported as environment variables or writing the resolved credentials to a file before executing Runway. Some users have created a script that handles this and would need to be run before Runway such as this one

Other IaC tools such as serverless are experiencing similar issues.

Describe the solution you'd like

Embed a dev build of botocore that supports AWS SSO profiles or, patch a AWS SSO credential provider into the general release of botocore.

When Runway is executed, if a AWS_PROFILE is set, use the embedded or patched botocore to retrieve the credentials and push them to the environment/context.env_vars (replacing AWS_PROFILE). This would enable all IaC tools that Runway wraps to take advantage of the AWS SSO credentials.

[REQUEST] drop python 3.5 support

Like python2, support for 3.5 is increasingly being left behind by runway's dependancies. The must recent being zipp, a dependency of importlib-metadata.

We should raise or new minimum required version to at least 3.6 which is generally more supported.

We have already stopped testing for 3.5, the next step is to modify the required python version in setup.py.

Request: `.flake8` configuration file support

Lines 112 - 114 in runway/commands/base.py are executing a flake8 run during runway preflight specifying exclude directives. Because of this, the .flake8 configuration file I have in the root of my repo is ignored.

I've verified this at the root of a repo I'm working in by manually executing flake8, both with --exclude=node_modules,.serverless and without. When I include the exclude directive, my .flake8 config file is ignored.

Can we check the filesystem for .flake8 (and maybe the other names flake8 supports) and source them if present? It seems odd we support overriding the yamllint configuration, but not flake8.

[REQUEST] There should be a “dev” deployment of auth@edge that does not use CloudFront

CloudFront takes forever to update making it a horrible dev experience when making iterative changes while troubleshooting. We should find to retain the full functionally of auth@edge but, without using CloudFront for faster iterations.

[TODO] refactor `runway.module.serverless.deploy_package` to be part of the `Serverless` class

#268 broke up the logic of that was once in one method into multiple methods. This created many methods that duplicate logic contained in deploy_package and some of the functions it calls. By moving the logic into methods on the Serverless class, it will reduce the overall code that will need to be maintained.

In performing this refactor and writing the appropriate tests, the runway/module/serverless.py should have 100% coverage to consider this done.

Pylint/Astroid Pipfile dependencies should be updated

The root Pipfile has pylint & astroid pinned back because of this bug.

Now that an updated astroid has been released with a fix, it should be possible to remove astroid from the Pipfile & loosen the pylint version

[REQUEST] Auth@Edge Static Sites check_auth should refresh tokens prior to expiry

The existing check for expiry is met at the time of expiration.

The AWS example has the refreshing occur 5 minutes prior to expiry. Assuming we wanted to follow the same logic, the change for us would look like:

if now > exp_date and refresh_token

to

if now > (exp_date - 5 minutes) and refresh_token

React sample app should be updated to not use yarn

Other nodejs examples only use npm, reducing our external dependencies.

Easiest option to get the new files is probably to recreate the sample using the --use-npm flag

Typescript samples should not include tslint

E.g. gen-sample sls-tsc

TSLint has been deprecated in favor of eslint. Any tslint examples/use should be converted to eslint directly.

Add `runway info` command

Background
Most of the deployment tools that runway supports out of the box have some way to provide a summary of the current state of the deployment.

Proposal
Add an info command to Runway (aliased to show) which in turn issues the following commands for modules:

• Terraform: terraform show
• Serverless: serverless info
• Cloudformation: stacker info

Static site and CDK modules will output a 'non-supported' error message.

[REQUEST] Overlay configuration for remote modules

What I'd like to be able to do is use a Serverless Framework runway remote module (e.g. this simple-http-endpoint) with some additional customization of the module.

E.g., today I can do:

deployments:
    - modules:
        - path: git::git://github.com/serverless/examples.git//examples/aws-node-simple-http-endpoint

But there's no clear way to add additional customizations to that project. Specifically, something like adding a new CFN resource in that directory's serverless.yml.

For CloudFormation modules, package_sources (when using the configs: option) enables this by way of yaml merging. That level of functionality isn't necessarily needed here, I'd be happy if local files could just be copied over the files in the remote repo.

Terraform: Support of YAML Dictionary to Go Map

Runway's transformation of yaml dict to Go map explodes.

tags:
  key1: value1

Error message:

filename = fspath(filename)  # Does type-checking of `filename`.
TypeError: expected str, bytes or os.PathLike object, not dict

Runway used to make it to at least the -tfvars tags = {'key1': 'value1' } ` step
A minimal example can be found here: https://github.com/agasperonica/terraform-list-bug-example

Package Versions

• Runway: 1.1.0
• Terraform v0.12.10

Operating System Info:

19.0.0 Darwin Kernel Version 19.0.0: 
root:xnu-6153.41.3~29/RELEASE_X86_64 x86_64

[REQUEST] escape runway.yml lookup syntax

In some cases, there is a need to pass a parameters or options in a format that looks like our lookup syntax (e.g. ${<something>}). There needs to be a way to escape the lookup, causing it to pass the literal value instead of resolving it.

Possible Syntax

• {{<something>}} in the runway or CFNgin config would translate to ${<something>} after lookups have been resolved
• #{<something>} in the runway or CFNgin config would translate to ${<something>} after lookups have been resolved

Should also work in CFNgin config files.

Static Site Auth At Edge cleanup stack does not use IAM role boundaries

The staticsite_role_boundary_arn parameter is used in all stacks but the final -cleanup stack. The IAM roles in that stack should have the same role boundary applied to them.

[BUG] serverless module does not respect `environments`

Describe the bug

Runway version 1.7.3

deployments:
  - modules:
      - path: something.sls
        environments:
          example: true

Using the above runway.yml example with no environment file in the module directory with a deploy environment of example, the module will be skipped even though it is explicitly enabled.

Expected behavior

The module should not be skipped.

This should be a quick fix, changing the logic here

Disable terminal color output automatically

CodeBuild (and obviously other systems) log output can be difficult to read with color formatting characters.

We should (for Terraform & Serverless, at least, where I know there are compatible options) automatically invoke modules with --no-color flags (I like @ITProKyle 's idea of modeling it after colorama).

An environment variable to explicitly override this option would be a nice to have too, since there will always be exceptions to this (also a great idea by @ITProKyle )

[Request] option to exclude paths from stacker blueprint tests

Problem

When using runway test, all python files in directories that have a suffix of blueprints are run an output checked for valid cloudformation. If this is run from the top level of a repo to test everything contained in it, there is no way to prevent it from running these tests on a handful of directories if they contain code that should be excluded for some reason.

Proposed Solutions

• add a config file to control the behavior of runway test. Could also be included in the runway.yml file itself.
• add cli arguments to exclude/ignore paths. runway test --ignore-path path/to/dir

[BUG] runway does not work with cfn-lint>=0.29.4

This version introduced a new positional argument resulting in the following error.

INFO:runway:======= Running test 'cfn-lint' ===========================
Traceback (most recent call last):
  File "/Users/person/.pyenv/versions/3.8.2/lib/python3.8/site-packages/runway/commands/runway/test.py", line 71, in execute
    handler.handle(test.name, test.args)
  File "/Users/person/.pyenv/versions/3.8.2/lib/python3.8/site-packages/runway/tests/handlers/cfn_lint.py", line 73, in handle
    exit_code = cls.run_cfnlint(args.get('cli_args', ''))
  File "/Users/person/.pyenv/versions/3.8.2/lib/python3.8/site-packages/runway/tests/handlers/cfn_lint.py", line 44, in run_cfnlint
    cfnlint.core.run_cli(  # pylint: disable=no-value-for-parameter
TypeError: run_cli() missing 1 required positional argument: 'build_graph'
ERROR:runway:Test failed: cfn-lint

will likely need to add a requirement of cfn-lint>=0.29.4 to runway.

The positional arg was added to line 25 of src/cfnlint/__main__,py in v0.29.4.

Static Site Auth At Edge fails to deploy from single binary builds

The static site module type invokes the cfngin aws_lambda hook, which picks up runway-cli as the python executable and raises an exception when it tries to run runway-cli -m pip ...

Error occurred for me on a Ubuntu 20.04 system with npm-installed/ran Runway v1.7.3.

$ npx runway deploy
<normal output through the build of the static site>

INFO:runway.hooks.staticsite.build_staticsite:staticsite: archiving app at <repo-root>/frontend to s3://<namespace>-frontend-dependencies-artifacts-1x8zky2a7xftl/<namespace>-frontend-c37190d638ef359009f0b6c8020dffa7.zip
INFO:runway.cfngin.hooks.aws_lambda:lambda: using custom bucket: <namespace>-frontend-dependencies-artifacts-1x8zky2a7xftl
INFO:runway.cfngin.hooks.aws_lambda:lambda: using requirements.txt for dependencies
ERROR:runway.cfngin.hooks.aws_lambda:"<repo-root>/node_modules/@onica/runway/src/runway/runway-cli -m pip install -t /home/troyready/.runway_cache/cfnginnkuld5kf -r /home/troyready/.runway_cache/cfnginnkuld5kf/requirements.txt" failed with the following output:
Usage:
  runway (test|preflight)
  runway (plan|taxi) [--tag <tag>...]
  runway (deploy|takeoff) [--tag <tag>...]
  runway (destroy|dismantle) [--tag <tag>...]
  runway init
  runway gen-sample [<samplename>]
  runway whichenv
  runway envvars
  runway run-aws <awscli-args>...
  runway run-python <filename>
  runway run-stacker <stacker-args>...
  runway tfenv (install|run) [<tfenv-args>...]
  runway kbenv (install|run) [<kbenv-args>...]
  runway -h | --help
  runway --version

ERROR:runway.cfngin.hooks.utils:Method runway.hooks.staticsite.auth_at_edge.lambda_config.write threw an exception:
Traceback (most recent call last):
  File "site-packages/runway/cfngin/hooks/utils.py", line 94, in handle_hooks
  File "site-packages/runway/hooks/staticsite/auth_at_edge/lambda_config.py", line 131, in write
  File "site-packages/runway/cfngin/hooks/aws_lambda.py", line 960, in upload_lambda_functions
  File "site-packages/runway/cfngin/hooks/aws_lambda.py", line 710, in _upload_function
  File "site-packages/runway/cfngin/hooks/aws_lambda.py", line 536, in _zip_package
runway.cfngin.exceptions.PipError: A non-zero exit code was returned when invoking pip. More information can be found in the error above.
Traceback (most recent call last):
  File "runway-script.py", line 2, in <module>
  File "site-packages/runway/cli.py", line 91, in main
  File "site-packages/runway/commands/modules/deploy.py", line 55, in execute
  File "site-packages/runway/commands/modules_command.py", line 371, in run
  File "site-packages/runway/commands/modules_command.py", line 452, in _process_deployments
  File "site-packages/runway/commands/modules_command.py", line 474, in _execute_deployment
  File "site-packages/runway/commands/modules_command.py", line 511, in _process_modules
  File "site-packages/runway/commands/modules_command.py", line 594, in _deploy_module
  File "site-packages/runway/module/staticsite.py", line 76, in deploy
  File "site-packages/runway/module/staticsite.py", line 110, in _setup_website_module
  File "site-packages/runway/module/cloudformation.py", line 18, in deploy
  File "site-packages/runway/cfngin/cfngin.py", line 124, in deploy
  File "site-packages/runway/cfngin/actions/base.py", line 162, in execute
  File "site-packages/runway/cfngin/actions/build.py", line 510, in pre_run
  File "site-packages/runway/cfngin/actions/build.py", line 185, in handle_hooks
  File "site-packages/runway/cfngin/hooks/utils.py", line 94, in handle_hooks
  File "site-packages/runway/hooks/staticsite/auth_at_edge/lambda_config.py", line 131, in write
  File "site-packages/runway/cfngin/hooks/aws_lambda.py", line 960, in upload_lambda_functions
  File "site-packages/runway/cfngin/hooks/aws_lambda.py", line 710, in _upload_function
  File "site-packages/runway/cfngin/hooks/aws_lambda.py", line 536, in _zip_package
runway.cfngin.exceptions.PipError: A non-zero exit code was returned when invoking pip. More information can be found in the error above.
[54165] Failed to execute script runway-script

Not sure exactly what the best path forward is. From what I can tell, it used to be supported to run pip from in python but that reportedly doesn't work anymore. If there was some way to do it, in binary packaging we control everything so maybe it's possible via another condition here -- changing it to runway-cli run-python ...

Interactive plan asking about destroying stacks

While testing runway plan today (v1.7.3 on python2 in a ubuntu:bionic docker image), I got unexpected output:

root@03c24adf8758:/app# export DEPLOY_ENVIRONMENT=prod
root@03c24adf8758:/app# runway plan  
	INFO:runway:Could not find runway.variables.yml or runway.variables.yaml in the current directory. Continuing without a variables file.
INFO:runway:
INFO:runway:Environment "prod" was determined from the DEPLOY_ENVIRONMENT environment variable.
INFO:runway:If this is not correct, update the value (or unset it to fall back to the name of the current git branch or parent directory).
INFO:runway:

Configured deployments:
<my deployment list here>


Enter number of deployment to run (or "all"): 2
INFO:runway:
INFO:runway:Found 1 deployment(s)
INFO:runway:
INFO:runway:
INFO:runway:======= Processing deployment 'mydeployname' ===========================
INFO:runway:
INFO:runway:Attempting to deploy 'prod' to region(s): us-east-1
INFO:runway:
INFO:runway:======= Processing region us-east-1 ===========================
INFO:runway:Verified current AWS account matches required account id 123456789012.
INFO:runway:
INFO:runway:---- Processing module 'pipeline_user.cfn' for 'prod' in us-east-1 --------------
INFO:runway:Module options: {'name': 'pipeline_user.cfn', 'parameters': {'permitted_role_arns': 'somerolearn', 'namespace': 'prod'}, 'tags': {}, 'class_path': None, 'env_vars': {}, 'environments': {}, 'environment': True, 'child_modules': [], 'path': 'pipeline_user.cfn', 'type': None, 'options': {}}
INFO:runway.cfngin:stacks.yaml: generating change sets...
INFO:runway.cfngin:Using interactive AWS provider mode.
INFO:runway.cfngin.actions.diff:Diffing stacks: teststack
INFO:runway.cfngin.providers.aws.default:prod-teststack changes:
Parameters Added: TableName

Changes:
- Add TerraformLockTable (AWS::DynamoDB::Table)
- Add TerraformStateBucket (AWS::S3::Bucket)
Show full change set? [y/n] n
Destroy stack 'prod-teststack'? [y/n] n
INFO:runway.cfngin.ui:teststack: skipped (canceled execution)

(the "Destroy stack 'prod-teststack'?" question being the strange part)

The cfngin config defines the stack fairly simply (with no option set to persist the stack graph):

namespace: ${namespace}
cfngin_bucket: ""

sys_path: ./

stacks:
  teststack:
    template_path: test_templates/tf_state.yaml
    variables:
      TableName: ${ssm /infra/${namespace}-troytest::default=defaulttablename}

When I ran the plan, no stack was present (the stack had previously been deployed and deleted).

EDIT: I realize this is probably much more expected behavior than was originally apparent to me. Makes sense to query the user about deleting the stack that I know see in the "REVIEW_IN_PROGRESS" state.

Fine if we want to just close this, but perhaps there's still a UX improvement that could be made here that warrants leaving it open.

[Request] upgrade requirement on boto3 to latest version

I need to create a custom lookup that uses some new boto3.client('ec2') functionality like describe_transit_gateways. It would be nice if runway shipped with the latest version of boto3 !

[Request] print info message when unknown directives are in the runway.yml

In an effort to avoid human errors, I suggest that we print warning messages when unknown directives in the yaml config to show that a deployment was potentially incorrectly configured.

The long version of this story:
A user was setting up a runway deployment from scratch that would be executed as that user and assume role into the proper account to perform deployments. They were running into problems with 403 access denied errors on their initial deployment test. Being that the 403 was the only error shown, they spent several hours (pulling in other engineers as well) to figure out what was going on and digging into cloudtrail to no avail. unfortunately, the issue was not a 403 access denied on the role being assumed, but an issue with runway not assuming the role.

Eventually, it was noticed that the message INFO:runway:Assuming rolewas not in the output, so there was something wrong with that part of the config. Turns out human error had a directive of assume_role instead of assume-role with a cost of around 10 man hours until someone noticed this simple typo.

Can we have Runway print an info message (or warn) stating something like this:

WARN:runway:Unknown directive found in runway.yml: assume_role. Please check the online documentation for valid values.

Thanks!

"deployments" could optionally have a name

Background
Runway projects consist of one or more deployments, each of which contains one or more modules.

Every module has a "name", in that it has a path string defined (either provided explicitly, or assumed to be the name of the module's folder), and we can use this in logs:

INFO:runway:---- Processing module 'api.sls' for 'dev' in us-east-1 --------------

Deployments have nothing to name them, though. So when we refer to them, we use their index in the list:

INFO:runway:Found 1 deployment(s)
INFO:runway:
INFO:runway:======= Processing deployment 0 ===========================

Proposal
A deployment will have an optional name which can be used in place of the index for display purposes:

INFO:runway:======= Processing deployment `global_resources` ==============

The particular name of the value in the YAML file code be... name? label?

___
deployments:
  - label: global_deployments
    modules:
    ....

[REQUEST] runway file lookup

This will be an upgraded version of the CFNgin file lookup, similar to the SSM lookup.

Acceptance Criteria

• can be used in both the runway.yml and CFNgin config files
• supports the syntax of the current CFNgin file lookup
• supports the new style syntax
• path should be relative to the config file the lookup is being used in (or retain the functionality of the CFNgin lookup if it functions differently)

[REQUEST] feature

Is your feature request related to a problem? Please describe.

I'm always frustrated when deployment jobs take 2x as long as they need to.

Describe the solution you'd like
in the runway.yml (or the driver config). the "- regions:" should have the ability to run in parallel

Describe alternatives you've considered

writting another wrapper around runway (seems very redundant to write a wrapper around a wrapper for a wrapper).

[TODO] restructure static site docs

Mimic the other module docs

• create directory named after the module type
• create three files in the new directory: configuration.rst, directory_structure.rst, advanced_features.rst
• (optional) create an examples.rst file in the directory
• move the contents of the module_configurations file for the module type to these new files and delete the old file. some minor rewrite may be required.
• add a section to index.rst with a toctree for the new addition.

Improve error message when shelling out to commands that don't exist

When commands are executed (e.g. yarn build specified as the build_steps of a staticsite), a stacktrace is generated if the command isn't found.

This is expressed as a subprocess FileNotFoundError exception, which doesn't point the user to the real problem ("You told me to run yarn build but you don't have yarn installed").

We should catch the exception and print a more helpful error message (noting the failed command) before exiting 1.

[REQUEST] Support for terraform's `parallelism`

Is your feature request related to a problem? Please describe.
While parallelism is an advanced feature of terraform it does have it's uses and I'd like to be able to speed up some of the deployments.

Describe the solution you'd like
A key specified in the module definition stating parallelism.

deployments:
  - modules:
      - name: example
        path: dynamic-uat.tf
        parallelism: 10

I'm not tied to this format, there may be better ways such as global.

Describe alternatives you've considered
I can manually run terraform and use parallelism.

[QUESTION] Will Runway support Terragrunt?

Question
I am just getting familiar with Runway, and see that it can be used with Terraform. Are there any plans to make Runway work with Terragrunt?

Context
I like the DRY structure I have with Terragrunt, so I thought I'd check.

Cannot Dismantle if stack is missing

Suppose we have runway configured for the following modules:

• Stack A
• Stack B
• Stack C

And Stack C go removed manually

runway dismantle

runway fails because it detects that Stack C doesn't exist.

i'll get an error:

(node:20148) ExperimentalWarning: The fs.promises API is experimental
 
  Serverless Error ---------------------------------------
 
  Stack 'stack-C-staging' does not exist
 
  Get Support --------------------------------------------
     Docs:          docs.serverless.com
     Bugs:          github.com/serverless/serverless/issues
     Issues:        forum.serverless.com
 
  Your Environment Information -----------------------------
     OS:                     darwin
     Node Version:           10.1.0
     Serverless Version:     1.32.0

Expected behavior is that it a stack is missing, it should continue dismantling the other stacks

[TODO] restructure kubernetes docs

Mimic the other module docs

• create directory named after the module type
• create three files in the new directory: configuration.rst, directory_structure.rst, advanced_features.rst
• (optional) create an examples.rst file in the directory
• move the contents of the module_configurations file for the module type to these new files and delete the old file. some minor rewrite may be required.
• add a section to index.rst with a toctree for the new addition.

Need Support to Specify different Serverless Config file

Serverless has a cli option to specify a serverless config file

serverless deploy -c serverless2.yml

wondering if there is a way to support this