versent / golang-lambda-pipeline

A sample golang serverless application, with multi-account CI/CD via CodePipeline

Golang Lambda Pipeline

This repository contains a serverless application, and multi-stage pipeline to perform CI/CD on it.

Setup

1. Launch the deploy role template to the deploy accounts, passing the build account's id.
2. Gather the cross-account role ARNs from the stacks' outputs.
3. Deploy the pipeline and its dependencies template to the build account, passing the deploy role ARNs and other parameters.
4. Get the repository URL from the pipeline's stack outputs.
5. Push this repository's code to the repository to trigger a build and deployment to the staging account.
6. Manually approve the deployment in to the production account.
7. Profit!

Accounts

Multiple accounts are used so that:

• Blast radius is limited. Issues in the deploy accounts cannot impact other deploy accounts.
• No runtime dependencies on the pipeline. The pipeline resources (and the associated resources) can be removed without impacting the running application.

While multiple accounts are part of the architecture, nothing is stopping you from using less accounts - just supply the same Account Id in multiple locations (e.g. when the pipeline and repo are in the production account). The same deployment role can be used for both the staging and production stages, and they can all be in the same account.

Recommended

The recommend deployment pattern is a "pipline in production" approach This is based on the reasoning that anything that can modify production, is production:

Alternative

Some setups involve the idea of a "build" environment, which is a production-like environment that is separate from the application's production environment (usually for administrative reasons).

This approach can also work, as long as appropriate controls are in place on the build environment:

Application

SAM Template

A serverless application (using AWS SAM) template is included to simulate a real-world application. The application consists of a golang Lambda function (with tests), and API Gateway endpoint.

The application's build is managed via a CodeBuild buildspec.yaml file. All of the tasks are managed via make.

Deploy Accounts

The accounts where the application will be deployed (e.g. staging and production). These accounts must have some resources deployed in them to facilitate the pipeline's access to deploy (e.g. cross-account role, etc).

• Changes are deploy to the same named stacks e.g. ApplicationName-staging.
• Changes are automatically applied to the staging stack.
• Manual approval is required before a change set is created in production, and before the change set is applied (i.e. there are two approvals required).

Design

Pipeline Stages

Build Pipeline

This high-level diagram represents the resources that are created by the dependencies-tempalate.yaml.

1. Developers push code.
2. Pipeline is triggered by commits to master.
3. A build container builds the source using make commands (defined in buildspec.yaml).
4. Build artefacts are packaged and stored for deployment in S3 using aws cloudformation package.
5. The change set is automatically deployed to the staging environment stack.
6. The change must be manually approved before anything is deployed to production.
7. The change set is applied to the production stack, and must then be approved again before it is deployed.

Deploy Account

The following resources are created and referenced in the deploy accounts:

Links

• Golang support for AWS Lambda,

• Building CI/CD Pipelines for Serverless Applications - SRV302 - re:Invent 2017 video

• Building CI/CD Pipelines for Serverless Applications - SRV302 - re:Invent 2017 slides

• All CodeBuild Docker images are on GitHub

golang-lambda-pipeline's Issues

Include bucket names in dependencies/pipeline stack outputs

'nuff said.

Split dependencies template in to layers?

Would this be too complicated as a starting point/demo application + pipeline? It would add extra deployment steps.

Specifically: splitting out the buckets would enable the cross-account roles to specifically mention them, further limiting the access they require.

Set parameters in application stack

Make the example a bit more real-world by setting parameters in the deploy action.

This could be via a parameters file or overrides.

Add notifications for approvals

Has already been started, but things are commented-out in the template see below):

• Create a SNS topic
• Subscribe a given email address to the topic
• Give PipelineRole permission to publish

Setup aws-sam-local support

While there are some recipes started in the makefile, they need to be finished and documented.

Run CodeBuild container locally

Is this worth the effort? If the build process leverages too many (any?) remote services, this might not be practical/valuable...

Add recipes to makefile so that buildspec.yaml can be run locally in a simulate CodeBuild environment.

Use the official CodeBuild images.

Support GitHub repos (public and private)

Not just CodeCommit.

Tag application stack

Tag stacks deploy via the pipeline using a template configuration file.

Setup S3 SSE encryption for artefacts and deployments

All buckets should use SSE.

• Create an KMS key
• Add encryption to build artefacts
• Configure key policy so that cross-account roles can use it to decrypt when doing deployments

Verify cross-account deployments work

I'm 99% sure they will, but I've only tested this in versent-innovation-2018 so far (i.e. the deploy role is in the same account).