aws-samples / aws-aurora-cloudformation-samples

Quick start reference deployment of an Amazon Aurora DB Cluster with AWS recommended best practices for security and high availability using AWS CloudFormation

AWS Aurora CloudFormation Samples

This GitHub repository contains:

1. A set of AWS CloudFormation samples to deploy an Amazon Aurora DB cluster based on AWS security and high availability best practices.
2. Python based source code to bootstrap the database upon creation using AWS Lambda.

When you are starting your journey with Amazon Aurora and want to set up AWS resources based on the recommended best practices of AWS Well-Architected Framework, you can use the CloudFormation templates provided here.

Deploy an Amazon Aurora PostgreSQL DB Cluster

Architecture overview

Here is a diagram of our architecture and a brief summary of what you are going to set up.

The sample CloudFormation templates provision the network infrastructure and all the components shown in the architecture diagram. I broke the CloudFormation templates into the following three stacks.

1. CloudFormation template to set up VPC, subnets, route tables, internet gateway, NAT gateway, S3 gateway endpoint, AWS Secrets Manager interface endpoint, and other networking components.
2. CloudFormation template to set up an Amazon Linux bastion host in an Auto Scaling group to connect to the Aurora PostgreSQL DB cluster.
3. CloudFormation template to set up Aurora PostgreSQL DB cluster with master user password stored in AWS Secrets Manager and bootstrap the database using AWS Lambda.

The stacks are integrated using exported output values. Using three different CloudFormation stacks instead of one nested stack gives you some flexibility. For example, you can choose to deploy the VPC and bastion host CloudFormation stacks once and Aurora PostgreSQL DB cluster CloudFormation stack multiple times in an AWS Region.

Best practices, Prerequisites and Set up Instructions

For Best practices incorporated in the sample AWS CloudFormation samples, prerequisites and set up instructions refer the following document. You can download this document and then launch AWS CloudFormation directly from it, by selecting the buttons embedded in the set up section.

Deploy an Amazon Aurora PostgreSQL DB cluster with recommended best practices using AWS CloudFormation

License

This library is licensed under the Apache 2.0 License.

aws-aurora-cloudformation-samples's Issues

Error when trying to deploy Aurora stack

Receiving the following error when deploying Aurora-Postgres-DB-Cluster.yml

CustomResource attribute error: Vendor response doesn't contain SecretRotationLambdaARN key in object arn:aws:cloudformation:us-east-1:XXXX:stack/stack-name/32785700-5b1d-11ed-a0a8-12aef090eaef|CreateSecretRotationLambdaFnTrigger|5f02622a-1de1-48ce-bba6-887eb762362c in S3 bucket cloudformation-custom-resource-storage-useast1

Error creating rds stack for EnvironmentStage = product

Hi, in our project, we are using as environment values:
EnvironmentStage: Type: String Description: The environment tag is used to designate the Environment Stage of the associated AWS resource. AllowedValues: - develop - staging - product Default: develop

For develop seems no problem, but when I try to create the product environment, the stack is giving error, can't create rotation lambda.

When I check cloudwatch log in /aws/lambda/SecretsManager-SecretRotationFn-stackname ,
I see this error:

[ERROR] ResourceNotFoundException: An error occurred (ResourceNotFoundException) when calling the GetSecretValue operation: Secrets Manager can't find the specified secret value for VersionId: 99542ad5-90bb-466a-b94b-d68e0c48f712

I think this error is happening because If I check logging before the error, It's creating secret with wrong name, It's putting prod instead of product in the name, but I'm not using prod in my template anywhere.

[INFO] 2021-01-06T09:02:53.425Z de4c8306-ab24-4594-8e87-82889466c6b9 createSecret: Successfully put secret for ARN arn:aws:secretsmanager:us-east-1:017212038965:secret:prod/aurora-pg/ep2as-product-rds-phqqY5 and version 99542ad5-90bb-466a-b94b-d68e0c48f712.

It's only happening for product, for develop It's working as expected.

For create the name, I'm obviously using same as the template, I only changed EnvironmentStage values of the template:

AuroraMasterSecret:
    Condition: IsNotUseDBSnapshot
    Type: AWS::SecretsManager::Secret
    Properties:
      Name: !Join ['/', [!Ref EnvironmentStage, 'aurora-pg', !Ref 'AWS::StackName']]
      Description: !Join ['', ['Aurora PostgreSQL Master User Secret ', 'for CloudFormation Stack ', !Ref 'AWS::StackName']]
      Tags:
        -
          Key: EnvironmentStage
          Value: !Ref EnvironmentStage
        -
          Key: DatabaseEngine
          Value: 'Aurora PostgreSQL'
        -
          Key: StackID
          Value: !Ref 'AWS::StackId'
      GenerateSecretString:
        SecretStringTemplate: !Join ['', ['{"username": "', !Ref DBUsername, '"}']]
        GenerateStringKey: "password"
        ExcludeCharacters: '"@/\'
        PasswordLength: 16

I guess I will be forced to use prod as EnvironmentStage, since I can't find the source code of the lambda SecretsManager-SecretRotationFn-stackname for fix this bug, but It would be nice If you could fix this bug.

VPC-SSH-Bastion.yml fails because of the bastion_bootstrap script 1.4 in the current AMI

The VPC-SSH-Bastion.yml template is broken with the latest version of bastion_bootstrap.sh script
The signal received from the AutoScalingGroup resource to CloudFormation is Failure because of the bastion bootstrap script.

If I search for errors in the bastion instance with the command:

grep -ni 'error\|failure' $(sudo find /var/log -name cfn\* -or -name cloud-init\*)

I can see:

/var/log/cfn-init.log:22:2021-12-30 16:21:16,428 [ERROR] Command b-bootstrap (REGION=eu-west-1 URL_SUFFIX=amazonaws.com BANNER_REGION=eu-west-1 ./bastion_bootstrap.sh --banner s3://aws-quickstart-eu-west-1/quickstart-linux-bastion/scripts/banner_message.txt --enable true --tcp-forwarding false --x11-forwarding false) failed
/var/log/cfn-init.log:43:2021-12-30 16:21:16,428 [ERROR] Error encountered during build of config: Command b-bootstrap failed

So the issue is in the UserData parameter in CloudFormation

This "sample" should be the gold standard for every AWS service

Kudos to this team!

This repo so fully implements the best practices of setting up RDS using CloudFormation, that all resource types should use it as a model.

As a customer of AWS and implementor of AWS on behalf of clients, I should be able to source complete public CloudFormation templates that implement the best practices of running a given infrastructure type.

I sorely wish I had a template repo exactly like this(auto-rotating passwords, VPCs, bastions, keys, permissions, everything) but for DocumentDB.