pauldevos / amazon-mwaa-examples

Amazon Managed Workflows for Apache Airflow (MWAA) Examples repository contains example DAGs, requirements.txt, plugins, and CloudFormation templates focused on Amazon MWAA.

Amazon Managed Workflows for Apache Airflow (MWAA) Examples

This repository contains example DAGs, requirements.txt, plugins, and CloudFormation templates focused on Amazon MWAA. Since Amazon MWAA is running open-source Apache Airflow many of the contributions will be applicable for self-managed implementations as well.

Contributing

We want examples of as many use cases in this repository as possible! Please submit a Pull Request if you would like to add something.

DAG Files

Example Directed Acyclic Graph (DAG) workflows that have been tested to work on Amazon MWAA. Associated requirements.txt will be referenced to the entry in the next section. See Amazon MWAA documentation for details.

• Move your Apache Airflow Connections and Variables to AWS Secrets Manager
• Amazon Managed Workflows for Apache Airflow (MWAA) and Amazon EMR
• Interactive Commands with Amazon MWAA and Bash Operator

Use Cases

This folder contains complete set of sample use cases including documentation, infrastructure as code, and dependant resources. Follow the README.md in each use case to get started.

• Image Processing Pipeline
• Amazon MWAA with AWS CodeArtifact for Python dependencies

Requirements.txt

Sometimes getting the right combination of Python libraries is tricky. This repository is here to help. See Amazon MWAA documentation for details.

• Amazon Backport Providers
• GCP Backport Providers

Plugins

Most Airflow community plugins will work fine on Amazon MWAA. This repository is for specific examples that have been testing on the service. See Amazon MWAA documentation for details.

CloudFormation

Example CloudFormation templates for Amazon MWAA. See AWS CloudFormation documentation for details.

IAM Policies

Example AWS IAM Policy definitions. See Amazon MWAA documentation for details.

Setup for the examples

These examples all assume a working Amazon MWAA environment.

Security

See CONTRIBUTING for more information.

License

This library is licensed under the MIT-0 License. See the LICENSE file.