zbrookle / avionix_airflow

Apache Airflow Hosted on a Kubernetes cluster, ready out of the box, with monitoring stack included (Grafana, ElasticSearch, Filebeat)

Avionix Airflow

Why are there so many guides on how to set up Apache Airflow when the code could just be given to you?!

Welcome to Avionix Airflow, an out of the box solution to installing airflow on Kubernetes.

Currently supported platforms are Minikube and AWS EKS (managed nodegroups only), pull requests adding support for GKE and AKS are welcome.

Overview of Avionix Airflow Capabilities

Avionix airflow provides the following out of the box solutions for airflow

1. Airflow Webserver and Scheduler with configured RBAC for the KubernetesExecutor
2. Airflow metric collection (statsd -> telegraf -> elasticsearch)
3. Airflow log collection (stdout -> filebeat -> elasticsearch)
4. Metric and log visualization (elasticsearch & postgres -> grafana)

Installation

pip install avionix_airflow

Requirements

• kubectl
• helm

Configuration

Avionix airflow is configured by using Options objects, there are currently 4 different objects:

1. AirflowOptions
2. PostgresOptions
3. MonitoringOptions
4. CloudOptions (Specified using LocalOptions or AwsOptions)
5. RedisOptions (Only for use with CeleryExecutor, *not recommended*)

These options are then passed into the function get_chart_builder, which can be used to retrieve an avionix chart builder object. For instruction on how to use the builder object, see avionix

FAQ

• How do I change the grafana dashboard?

  • You can change the dashboard by setting the MonitoringOptions role to "Admin"

Documentation

Additional documentation can be found here

avionix_airflow's Issues

need way to mount efs on external storage

Add built in monitoring that can be accessed via ingress

Needs external monitoring for memory of Kubernetes, maybe grafana?

Add option to add domain to os if not present

Make into publishable package

Add in external storage for additional write space on docker images

Get output of all helm messages and redirect to info so that they don't display in stdout

Allow customization for load balancers

Add readme usage documentation

Add internal health checks on services

Add remote logging options

Store credentials in secret variables

Add architecture diagram

ENH: Add id_rsa

Add ability to mount secret id_rsa onto each worker from a secret (for programs that need to use git or some other remote access)

a

ENH: Create landing page that base url will land on

Make worker pod dag folder read only

Add in way to mount ebs as the source for logs/dags

Add in generation of fernet key on creation

Add file beats for piping logging from all machines into elasticsearch

Add pipeline

Add Kubernetes dashboard