ran-isenberg / aws-lambda-handler-cookbook

This repository provides a working, deployable, open source-based, serverless service template with an AWS Lambda function and AWS CDK Python code with all the best practices and a complete CI/CD pipeline.

AWS Lambda Handler Cookbook (Python)

This project provides a working, open source based, AWS Lambda handler skeleton Python code including DEPLOYMENT code with CDK and a pipeline.

This project can serve as a template for new Serverless services - CDK deployment code, pipeline and handler are covered.

📜Documentation | Blogs website

Contact details | [email protected]

Getting Started

You can start with a clean service out of this template repository without using the 'Template' button on GitHub.

You can use Cookiecutter.

• Cookiecutter - install with pip/brew brew install cookiecutter or pip install cookiecutter

Then run:

cookiecutter gh:ran-isenberg/cookiecutter-serverless-python

Answer the questions to select repo name, service name, etc.:

That's it, your developer environment has been set! you are ready to deploy the service:

cd {new repo folder}
poetry shell
make deploy

You can also run 'make pr' will run all checks, synth, file formatters , unit tests, deploy to AWS and run integration and E2E tests.

The Problem

Starting a Serverless service can be overwhelming. You need to figure out many questions and challenges that have nothing to do with your business domain:

• How to deploy to the cloud? What IAC framework do you choose?
• How to write a SaaS-oriented CI/CD pipeline? What does it need to contain?
• How do you handle observability, logging, tracing, metrics?
• How do you write a Lambda function?
• How do you handle testing?
• What makes an AWS Lambda handler resilient, traceable, and easy to maintain? How do you write such a code?

The Solution

This project aims to reduce cognitive load and answer these questions for you by providing a skeleton Python Serverless service template that implements best practices for AWS Lambda, Serverless CI/CD, and AWS CDK in one template project.

Concepts

I spoke at AWS re:invent 2023 with Heitor Lessa, Chief Architect of Powertools for AWS Lambda about the concepts I implemented in this project.

Serverless Service - The Order service

• This project provides a working orders service where customers can create orders of items.

• The project deploys an API GW with an AWS Lambda integration under the path POST /api/orders/ and stores data in a DynamoDB table.

Monitoring Design

Features

• Python Serverless service with a recommended file structure.
• CDK infrastructure with infrastructure tests and security tests.
• CI/CD pipelines based on Github actions that deploys to AWS with python linters, complexity checks and style formatters.
• CI/CD pipeline deploys to dev/staging and production environments with different gates between each environment
• Makefile for simple developer experience.
• The AWS Lambda handler embodies Serverless best practices and has all the bells and whistles for a proper production ready handler.
• AWS Lambda handler uses AWS Lambda Powertools.
• AWS Lambda handler 3 layer architecture: handler layer, logic layer and data access layer
• Features flags and configuration based on AWS AppConfig
• Idempotent API
• REST API protected by WAF with four AWS managed rules in production deployment
• CloudWatch dashboards - High level and low level including CloudWatch alarms
• Unit, infrastructure, security, integration and end to end tests.
• Automatically generated OpenAPI endpoint: /swagger with Pydantic schemas for both requests and responses
• CI swagger protection - fails the PR if your swagger JSON file (stored at docs/swagger/openapi.json) is out of date
• Automated protection against API breaking changes

CDK Deployment

The CDK code create an API GW with a path of /api/orders which triggers the lambda on 'POST' requests.

The AWS Lambda handler uses a Lambda layer optimization which takes all the packages under the [packages] section in the Pipfile and downloads them in via a Docker instance.

This allows you to package any custom dependencies you might have, just add them to the Pipfile under the [packages] section.

Serverless Best Practices

The AWS Lambda handler will implement multiple best practice utilities.

Each utility is implemented when a new blog post is published about that utility.

The utilities cover multiple aspect of a production-ready service, including:

• Logging
• Observability: Monitoring and Tracing
• Observability: Business KPIs Metrics
• Environment Variables
• Input Validation
• Dynamic Configuration & feature flags
• Start Your AWS Serverless Service With Two Clicks
• CDK Best practices
• Serverless Monitoring
• API Idempotency
• Serverless OpenAPI Documentation with AWS Powertools

Getting started

Head over to the complete project documentation pages at GitHub pages at https://ran-isenberg.github.io/aws-lambda-handler-cookbook

Code Contributions

Code contributions are welcomed. Read this guide.

Code of Conduct

Read our code of conduct here.

Connect

• Email: [email protected]
• Blog Website RanTheBuilder
• LinkedIn: ranisenberg
• Twitter: IsenbergRan

Credits

• AWS Lambda Powertools (Python)

License

This library is licensed under the MIT License. See the LICENSE file.

aws-lambda-handler-cookbook's Issues

make deploy takes quite some time to complete.

Describe the bug
Time taken for 'make deploy' is very long.
Please try to remove the unwanted steps in the 'make' command to improve performance.

Also , Please state the prereqs clearly , as to what permissions are required before executing 'make deploy'.

I got the error This CDK deployment requires bootstrap stack version '6', but during the confirmation via SSM parameter /cdk-bootstrap/hnb659fds/version the following error occurred: AccessDeniedException: User: arn:aws:iam::262xxx2701:user/[email protected] is not authorized to perform: ssm:GetParameter on resource: arn:aws:ssm:ap-south-1:26xxxx822701:parameter/cdk-bootstrap/hnb659fds/version because no identity-based policy allows the ssm:GetParameter action

To Reproduce
Steps to reproduce the behavior:
execute 'make deploy'

Expected behavior

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

• OS: [e.g. iOS]
• Browser [e.g. chrome, safari]
• Version [e.g. 22]

Smartphone (please complete the following information):

• Device: [e.g. mac]
• OS: [mac 14.1]

Additional context
Add any other context about the problem here.

ServiceStack initializer complains about too many positional args

Describe the bug
Deploy fails. The code looks fine to me though.

To Reproduce
Steps to reproduce the behavior:
Follow steps from https://ran-isenberg.github.io/aws-lambda-handler-cookbook/getting_started/ exactly. make deploy fails.

Expected behavior
make deploy works.

Screenshots

class ServiceStack(Stack):
    def __init__(self, scope: Construct, id: str, **kwargs) -> None:
        super().__init__(scope, id, **kwargs)

        # This construct should be deployed in a different repo and have its own pipeline so updates can be decoupled
        # from running the service pipeline and without redeploying the service lambdas. For the sake of this template
        # example, it is deployed as part of the service stack
        self.dynamic_configuration = ConfigurationStore(
            self,
            f"{id}dynamic_conf"[0:64],
            ENVIRONMENT,
            SERVICE_NAME,
            CONFIGURATION_NAME,
        )
        self.lambdas = ApiConstruct(
            self, f"{id}Service"[0:64], self.dynamic_configuration.config_app.name
        )

Desktop (please complete the following information):

• OS: Ubuntu 22.04

feature: CDK v2 Upgrade

The CDK deployment code is v1. V1 will not be supported in June.

feat: `cdk.json` could use `poetry run`

Is your feature request related to a problem? Please describe.

Currently cdk.json requires the cdk dependencies to be installed or run within poetry shell

Describe the solution you'd like

Changing the cdk.json to the following will allow cdk to run without extra streps.

{
  "app": "poetry run python app.py"
}

NOTE: pip install will no longer be needed in the github workflow

aws-lambda-handler-cookbook/.github/workflows/serverless-service.yml

Line 35 in 936588d

pip install -r dev_requirements.txt

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

Feature request: GitLab version

Is your feature request related to a problem? Please describe.
As a gitlab user, I'd like to use this cookbook.

Describe the solution you'd like
Preferably, the cookiecutter interface would prompt me for whether I'm in gitlab or in github, and implement the appropriate files.

Describe alternatives you've considered
I'm considering implementing it for myself, but there are many items that are specific to github actions that aren't available in gitlab yet.

Build errors related to "service" reference not being found

Describe the bug
When running cdk synth or make deploy I'm getting the following error:

make deploy
mkdir -p .build/lambdas ; cp -r service .build/lambdas
cp: service: No such file or directory
make: *** [build] Error 1

To Reproduce
Steps to reproduce the behavior:

1. Run cdk synth or make deploy
2. See error

Expected behavior
Expect it to actually synthesize or deploy

Screenshots
If applicable, add screenshots to help explain your problem.

Additional context
I believe this might be related to cookiecutter, will try to further troubleshoot and fix but hoping you'll know exactly what is going on before I do.

feat: use `npx cdk` to

Is your feature request related to a problem? Please describe.

Makefile is expecting cdk to be installed and have a compatible version with the project

Describe the solution you'd like

Use a package.json to specify the version of cdk needed and a script to run is

{
  "devDependencies": {
    "aws-cdk": "^2.78.0"
  },
  "scripts": {
    "cdk": "cdk"
  }
}

To keep the Makefile the same an alias for cdk can be made alias cdk="npx aws-cdk"

Describe alternatives you've considered

Documenting that an alias like this could also be used alias cdk="npx aws-cdk"

Additional context

• CDK tips, part 1 – how to use local CDK commands
• Using local tsc and cdk

New run of cookiecutter template not generating with latest changes from `main` branch including new `kwarg` keywords to `appconfig.{Application,Environment}`

Describe the bug
It looks like the latest version of the cookiecutter template is not pulling in recently merged changes. Having not written a cookiecutter template yet, I do not quite know why this would be the case; I expected cookiecutter gh:ran-isenberg/cookiecutter-serverless-python to get the latest code changes.

To Reproduce
Steps to reproduce the behavior:

1. run cookiecutter gh:ran-isenberg/cookiecutter-serverless-python, and select "delete and re-download it" upon being prompted (install should complete successfully)
2. open cdk/service/configuration/configuration_construct.py in your IDE and look at the appconfig.Application and appconfig.Environment lines
3. note that the changes introduced in #784 to update the initialization of the Application and Environment variables are not present (kwargs in the generated configuration_construct.py file includes name rather than application_name for example)
4. this will lead to an error upon make deploy: cdk TypeError: "Environment.__init__()" got an "unexpected keyword" argument "name" as well as the equivalent for AppConfig

Expected behavior

• running cookiecutter gh:ran-isenberg/cookiecutter-serverless-python should install the latest version of the origin/main branch, and make deploy should have the correct instantiation kwargs for appconfig.Application and appconfig.Environment

Desktop (please complete the following information):

➜  genai cookiecutter --version
Cookiecutter 2.5.0 from /Users/<uname>/.pyenv/versions/3.12.1/lib/python3.12/site-packages (Python 3.12.1 (main, Jan 29 2024, 13:55:08) [Clang 15.0.0 (clang-1500.1.0.2.5)])

• OS: Mac OS Ventura 13.6

Additional considerations

• I also tried cookiecutter gh:ran-isenberg/cookiecutter-serverless-python --checkout main, and for some reason, that, too had the older, outdated code. I am in the process of investigating why.

Additional question

• is there is an expected technique for pulling newly merged changes from the origin repository into a repository generated by cookiecutter? If so, a note of that in the README.md would be helpful for those who have already run the template and wish to pull in those changes. UPDATE - I see that copier is an alternative to cookiecutter that allows post-generation updates

Typo in cdk doc

Describe the bug
at line 30 of docs/cdk.md file the ASW should be AWS.

Align windows makefile to the mac variation

As I dont have a windows machine, i wasn't able to align the makefile to the windows makefile variation Makefile_windows.
Any help is appreciated to make sure all actions there are implemented and work.