Description
This repo contains implementations of Airflow workflows and tasks called respectively DAGs and Operators.
- DAGs - Direct Acyclic Graphs - Python scripts defining workflows in a way that reflects their relationships.
- Operators - Python functions which define the individual tasks that are executed as part of a DAG run.
To learn how to write DAGs and Operators read about core concepts and follow the official tutorial.
Examples
Simple working DAGs taken from Airflow documentation:
bash_commands.py
- Use ofBashOperator
and simple layout.task_decorator.py
- More complex layout withDummyOperator
.task_generator.py
- Semi-dynamic way to generate tasks.taskflow_api.py
-SimpleHttpOperator
andEmailOperator
.
These were researched in infra-bi#1. More examples are always welcome.
Continuous Integration
Changes pushed to master
are automatically fetched to our Airflow instance by the airflow-webhook
service.
Infrastructure
All Airflow infrastructure is managed in the infra-bi repository.