Learning by doing and thinking about the code we written.

This lab is designed to help you get familiar with Apache Airflow. You will learn how to create a simple DAG, schedule it, monitor its execution, and more.

I'd try to go beyond the basic concepts and cover some advanced topics like TaskGroup, TaskFlow API, SLA, Data-aware scheduling, Kubernetes Executor, and more.

Note: You can use Astro CLI to create a new Airflow project. For more information, see Astro CLI

• Basic knowledge of Python
  • Variables
  • Functions
  • Control Flow
  • arg and kwargs
• Basic knowledge of Docker
  • docker compose up and down is good enough
• Poetry: Package Manager for Python
  • poetry install --no-root to install dependencies
• Some parts of the lab require basic knowledge of Kubernetes and Helm
  • You can skip those parts if you are not familiar with them

1. Configuration

   • Lightweight Airflow setup with Docker, see docker-compose.lite.yaml
   • Enable Test button in Airflow UI
   • Disable Example DAGs
   • Copy Airflow Configuration
   • Enable Flower UI

2. What's Airflow?

   • Data Pipeline
   • Workflow Orchestration

3. Overview of Airflow UI and concepts

   • Airflow UI
     • Pause/Unpause
     • Trigger DAG
     • Refresh
     • Recent Tasks
     • DAG Runs
     • Graph View
   • DAGs
   • Operators
   • Tasks

4. Writing your first DAG (Single Operator)

   • Create a new DAG with PythonOperator
   • Defining DAG
     • Schedule
     • Task
   • Test the DAG

5. Writing your second DAG (Multiple Operators)

   • Create a new DAG with PythonOperator
   • Define dependencies between tasks
   • Test the DAG

6. Schedule your DAG

   • Fixed Interval
   • Cron Expression
   • Preset Airflow Scheduler

7. Google Drive to GCS

   • Create a new DAG
   • Create a new connection for Google Drive via Service Account
   • Use GoogleDriveToGCSOperator to copy files from Google Drive to GCS
   • Test the DAG

8. Working with Sensor

   • GoogleDriveFileSensor to wait for a file to be uploaded to Google Drive

9. Scraping Data from Githubs to Postgres

   • SimpleHTTPOperator to get data from Github API
   • PostgresOperator to insert data into Postgres

10. Trigger Other DAGs

    • Learn how to trigger another DAG
    • Getting to know TriggerDagRunOperator

11. Task Decorators - Taskflow API

    • Simplified way to define tasks
    • Getting to know @task decorator
    • Using @task to define taks like PythonOperator

12. Testing - In Progress

    • Unit Testing
    • DAG Integrity Testing
    • dag.test() method

13. Dataset - Data-aware scheduling - In Progress

    • Trigger DAG based on the data availability
    • Wait for many datasets to be available

14. Celery Executor (Local)

    • Monitor the task execution with Flower UI (To enable Flower UI, see chapter-0)
    • Add more workers to the Celery Executor
      • Duplicate airflow-worker service in docker-compose.yml and rename it
      • Restart Docker

15. Dependencies between Tasks

    • Basic define dependencies between tasks
    • Fan-in and Fan-out
    • Trigger Rules
    • Conditional Trigger

16. Managing Complex Tasks with TaskGroup

    • Group tasks together
    • Define dependencies between TaskGroups

17. SLA - Service Level Agreement

    • Define SLA for a DAG
    • Define SLA for a task
    • Define SLA callback

18. Airflow on Kubernetes - In Progress

    • Deploy Airflow on Kubernetes Cluster using Helm and Kind
    • Use Kubernetes Executor
    • Use KubernetesPodOperator

19. Build Airflow Docker Image

    • Build Airflow Docker Image (Poetry for Package Management)

20. Working with DataHub - In Progress

    • Setup DataHub on Local Development
    • Emit Metadata to DataHub