This repository contains common and extensible Prefect patterns to drive efficient workflows โ we like to call these patterns our recipes
Here you'll find starter code and more advanced example use cases.
We're always looking for new contributions! See our existing Recipe Ideas/Issues for inspiration. And check out Contributions to learn how you can add your Prefect 2.0 recipe and earn some swag!
To report issues, typos, or link fixes, please open an issue. We appreciate it!
What are you interested in seeing examples of? Jot down your big idea here.
- Getting Started With Subflows
- Run Multiple Subflows or Child Flows in Parallel
- Subflow with a Different Task Runner Than Parent Flow
- Create a Subflow and Block Until It's Completed
- Running Subflows On Their Own Infrastructure Using a Separate Deployment
- Conditionally Stop a Task Run
- Ensure Tasks Immediately Fail If Upstream Task Fails
- Define State Dependencies Between Tasks
- Triggering Flow Runs From a Deployment via API Call or From a Terminal Using Curl
- Event-Driven Data Pipelines with AWS Lambda and GitHub Actions
- Deploy Flows to Run as a Local Process, Docker Container or a Kubernetes Job
- Deploy Flows to AWS
- Deploy Flows to GCP
- Deploy Flows to Azure
- Store Flows in GitHub and Execute in a Docker Container
- Python-Based Deployments
- Serverless Real-Time Data Pipelines on AWS with Prefect, ECS and GitHub Actions
- Build a Data Platform with Prefect, dbt, and Snowflake (using blocks)
- Export Airbyte Configuration and Load to S3 bucket using blocks, including Python-based deployment
- ELT with Snowflake Using Async and Blocks
- Deploy a Prefect agent to ECS using the AWS CLI
- Deploy a Prefect agent to ECS with Terraform
- Deploy Flows Using ECSTask Infrastructure Blocks
- Deploy a Prefect agent to ECS Fargate using CloudFormation and GitHub Actions
- Setup Azure with Prefect
- Deploy Prefect Orion to an AKS Cluster with Azure Blob Storage
- Setup an Azure VM and Run the Prefect Agent
- Deploy Flows Using Azure Container Instances Infrastructure Blocks
- Build flow image and Prefect deployment with storage and infra Blocks on push to branch
- Build and Push flow docker image to Google Artifact Registry
- Build / Apply prefect deployment with blocks on change to python files
- Automate Prefect Deployments to AWS ECS Fargate Using GitHub Actions
- Automate Python-Based Deployments with GitHub Actions
- Register a Prefect Flow
- Run GraphQL Queries
- Airbyte Orchestration
- ETL with AWS S3 and Snowflake
- Use AWS Lambda for Event-Driven Flows
- Read Secrets into Prefect Cloud tenant
- Handle DBT Model Failures
- S3 Flow Storage on EKS
- Use LocalExecutor to run Dask computations on a Coiled cluster
- Use DaskExecutor to run Prefect tasks in parallel on a Coiled cluster
We use our Slack Community and Discourse to discuss all things Prefect-- such as FAQ, use cases and integrations. Join in the conversation ๐
We're always looking for new contributions! You can add your Prefect 2.0 recipe and earn some swag in a few simple steps:
- Look through the recipes to ensure your example is unique
- Clone the prefect-recipes repo:
git clone [email protected]:PrefectHQ/prefect-recipes.git
- Create and checkout a new branch:
git branch feat/recipe-name
git checkout feat/recipe-name
- Add your code under the appropriate category, making sure it is reproducible and easy to understand.
- Add your recipe to README.
- Commit and push the code to your remote branch.
- Create a PR ๐ค
Thank you for your contributions and efforts to improve prefect-recipes. We're glad to have you in our community!