Topic: airflow Goto Github

👇 Here are 2036 public repositories matching this topic...

abhishek-ch / around-dataengineering
airflow,A Data Engineering & Machine Learning Knowledge Hub
User: abhishek-ch
data-engineering machine-learning airflow datascience spark mlops devops infrastructure

airflow-helm / charts
airflow,The User-Community Airflow Helm Chart is the standard way to deploy Apache Airflow on Kubernetes with Helm. Originally created in 2017, it has since helped thousands of companies create production-ready deployments of Airflow on Kubernetes.
Organization: airflow-helm
Home Page: https://github.com/airflow-helm/charts/tree/main/charts/airflow
helm airflow kubernetes helm-charts helm-chart charts chart k8s

airflow-plugins / example-airflow-dags
airflow,Example DAGs using hooks and operators from Airflow Plugins
Organization: airflow-plugins
apache-airflow dag etl airflow-plugins selenium google-analytics facebook-ads sftp marketo salesforce hubspot marketo-sdk imap airflow mongodb mailgun singer dbt

ajbosco / dag-factory
airflow,Dynamically generate Apache Airflow DAGs from YAML configuration files
User: ajbosco
airflow apache-airflow dags python

alanchn31 / data-engineering-projects
airflow,Personal Data Engineering Projects
User: alanchn31
data-lake ingest-data data-engineering data-warehouse cassandra aws-redshift mongodb scrapy spark airflow

andreax79 / airflow-code-editor
airflow,A plugin for Apache Airflow that allows you to edit DAGs in browser
User: andreax79
apache-airflow airflow-plugin airflow python

andresionek91 / airflow-autoscaling-ecs
airflow,Airflow Deployment on AWS ECS Fargate Using Cloudformation
User: andresionek91
data-engineering airflow airflow-autoscaling-ecs airflow-deployment airflow-ecs

ankurchavda / streamify
airflow,A data engineering project with Kafka, Spark Streaming, dbt, Docker, Airflow, Terraform, GCP and much more!
User: ankurchavda
airflow kafka data-engineering python dbt spark gcp

apache / airflow
airflow,Apache Airflow - A platform to programmatically author, schedule, and monitor workflows
Organization: apache
Home Page: https://airflow.apache.org/
airflow apache apache-airflow python scheduler workflow automation dag data-engineering data-integration

apache / airflow-client-python
airflow,Apache Airflow - OpenApi Client for Python
Organization: apache
Home Page: https://airflow.apache.org/
airflow apache apache-airflow apache-airflow-client python

apache / dolphinscheduler
airflow,Apache DolphinScheduler is the modern data orchestration platform. Agile to create high performance workflow with low-code
Organization: apache
Home Page: https://dolphinscheduler.apache.org/
workflow-schedule azkaban airflow task-scheduler job-scheduler cloud-native data-pipelines orchestration workflow workflow-orchestration

apachecn / airflow-doc-zh
airflow,:book: [译] Airflow 中文文档
Organization: apachecn
Home Page: https://airflow.apachecn.org
airflow

argoproj / argo-workflows
airflow,Workflow Engine for Kubernetes
Organization: argoproj
Home Page: https://argo-workflows.readthedocs.io/
workflow kubernetes argo dag knative airflow machine-learning argo-workflows workflow-engine hacktoberfest

astronomer / airflow-chart
airflow,A Helm chart to install Apache Airflow on Kubernetes
Organization: astronomer
helm-chart airflow apache-airflow kubernetes

astronomer / astro-sdk
airflow,Astro SDK allows rapid and clean development of {Extract, Load, Transform} workflows using Python and SQL, powered by Apache Airflow.
Organization: astronomer
Home Page: https://astro-sdk-python.rtfd.io/
python sql pandas airflow sqlite bigquery postgres snowflake gcs s3

astronomer / astronomer-cosmos
airflow,Run your dbt Core projects as Apache Airflow DAGs and Task Groups with a few lines of code
Organization: astronomer
Home Page: https://astronomer.github.io/astronomer-cosmos/
airflow airflow-operators apache-airflow dbt python workflow

chandulal / airflow-testing
airflow,Airflow Unit Tests and Integration Tests
User: chandulal
airflow airflow-testing testing airflow-dags

coder2j / airflow-docker
airflow,Source code of the Apache Airflow Tutorial for Beginners on YouTube Channel Coder2j (https://www.youtube.com/c/coder2j)
User: coder2j
airflow airflow-docker tutorial docker-compose airflowtutorial docker

cubefs / compass
airflow,Compass is a task diagnosis platform for bigdata
Organization: cubefs
bigdata spark hadoop flink diagnose mapreduce scheduler sql airflow dolphinscheduler

damklis / dataengineeringproject
airflow,Example end to end data engineering project.
User: damklis
big-data scraping mongodb elasticsearch data-engineering kafka kafka-connect debezium django-rest-framework redis

elyra-ai / elyra
airflow,Elyra extends JupyterLab with an AI centric approach.
Organization: elyra-ai
Home Page: https://elyra.readthedocs.io/en/stable/
notebooks notebook-jupyter jupyterlab jupyterlab-extensions ai machine-learning pipelines kubeflow-pipelines python elyra

epoch8 / airflow-exporter
airflow,Airflow plugin to export dag and task based metrics to Prometheus.
Organization: epoch8
airflow airflow-exporter prometheus

feluelle / airflow-diagrams
airflow,Auto-generated Diagrams from Airflow DAGs. 🔮 🪄
User: feluelle
airflow diagrams auto-generated python cli

fieldryand / goflow
airflow,Simple but powerful DAG scheduler and dashboard
User: fieldryand
directed-acyclic-graph airflow dashboard go job-scheduler schedule workflow-engine

gocardless / airflow-dbt
airflow,Apache Airflow integration for dbt
Organization: gocardless
Home Page: https://pypi.org/project/airflow-dbt/
dbt airflow-dbt airflow

godatadriven / whirl
airflow,Fast iterative local development and testing of Apache Airflow workflows
Organization: godatadriven
airflow docker local-development testing

googlecloudplatform / airflow-operator
airflow,Kubernetes custom controller and CRDs to managing Airflow
Organization: googlecloudplatform
kubernetes kubernetes-operator apache-airflow airflow crd kubernetes-controller workflow-engine airflow-operator

hgrif / airflow-tutorial
airflow,Airflow basics tutorial
User: hgrif
airflow python tutorial

huseinzol05 / gather-deployment
airflow,Gathers Python deployment, infrastructure and practices.
User: huseinzol05
airflow docker docker-compose kafka pyflink pyspark python tensorflow

iusztinpaul / energy-forecasting
airflow,🌀 𝗧𝗵𝗲 𝗙𝘂𝗹𝗹 𝗦𝘁𝗮𝗰𝗸 𝟳-𝗦𝘁𝗲𝗽𝘀 𝗠𝗟𝗢𝗽𝘀 𝗙𝗿𝗮𝗺𝗲𝘄𝗼𝗿𝗸 | 𝗟𝗲𝗮𝗿𝗻 𝗠𝗟𝗘 & 𝗠𝗟𝗢𝗽𝘀 for free by designing, building and deploying an end-to-end ML batch system ~ 𝘴𝘰𝘶𝘳𝘤𝘦 𝘤𝘰𝘥𝘦 + 2.5 𝘩𝘰𝘶𝘳𝘴 𝘰𝘧 𝘳𝘦𝘢𝘥𝘪𝘯𝘨 & 𝘷𝘪𝘥𝘦𝘰 𝘮𝘢𝘵𝘦𝘳𝘪𝘢𝘭𝘴
User: iusztinpaul
Home Page: https://www.pauliusztin.me/courses/the-full-stack-7-steps-mlops-framework
airflow batch-processing feature-store github-actions mlops python cicd docker fastapi gcp

jghoman / awesome-apache-airflow
airflow,Curated list of resources about Apache Airflow
User: jghoman
airflow apache-airflow workflow-management awesome awesome-list

jgoerner / data-science-stack-cookiecutter
airflow,🐳📊🤓Cookiecutter template to launch an awesome dockerized Data Science toolstack (incl. Jupyster, Superset, Postgres, Minio, AirFlow & API Star)
User: jgoerner
data-science docker python jupyter superset airflow apistar postgres minio docker-image

josephmachado / beginner_de_project
airflow,Beginner data engineering project - batch edition
User: josephmachado
Home Page: https://www.startdataengineering.com/post/data-engineering-project-for-beginners-batch-edition/
airflow etl python docker emr redshift spark redshift-cluster database engineering

obenner / data-engineering-interview-questions
airflow,More than 2000+ Data engineer interview questions.
User: obenner
airflow avro aws azure cassandra data-engineering data-structures flink flume hadoop hadoop-hdfs hbase hive impala interview interview-questions kafka nifi spark sql

orchest / orchest
airflow,Build data pipelines, the easy way 🛠️
Organization: orchest
Home Page: https://orchest.readthedocs.io/en/stable/
data-science machine-learning pipelines ide jupyter cloud self-hosted jupyterlab notebooks docker

pipeline-tools / gusty
airflow,Making DAG construction easier
Organization: pipeline-tools
Home Page: https://pipeline-tools.github.io/gusty-docs/
airflow data-pipeline data-etl

pipelineai / pipeline
airflow,PipelineAI
Organization: pipelineai
Home Page: https://generativeaionaws.com
machine-learning artificial-intelligence tensorflow kubernetes cassandra spark kafka airflow docker redis

puckel / docker-airflow
airflow,Docker Apache Airflow
User: puckel
docker-airflow airflow docker scheduler workflow task management

pytorch / torchx
airflow,TorchX is a universal job launcher for PyTorch applications. TorchX is designed to have fast iteration time for training/research and support for E2E production ML pipelines when you're ready.
Organization: pytorch
Home Page: https://pytorch.org/torchx
pytorch machine-learning kubernetes slurm distributed-training pipelines components deep-learning python aws-batch

raystack / optimus
airflow,Optimus is an easy-to-use, reliable, and performant workflow orchestrator for data transformation, data modeling, pipelines, and data quality management.
Organization: raystack
Home Page: https://raystack.github.io/optimus
airflow etl workflows automation golang bigquery data-warehouse analytics data-modelling analytics-engineering

rjurney / agile_data_code_2
airflow,Code for Agile Data Science 2.0, O'Reilly 2017, Second Edition
User: rjurney
Home Page: http://bit.ly/agile_data_science
data-syndrome data data-science analytics apache-spark apache-kafka kafka spark predictive-analytics machine-learning

san089 / goodreads_etl_pipeline
airflow,An end-to-end GoodReads Data Pipeline for Building Data Lake, Data Warehouse and Analytics Platform.
User: san089
etl-pipeline etl-framework spark apache-spark apache-airflow airflow redshift emr-cluster livy s3

san089 / udacity-data-engineering-projects
airflow,Few projects related to Data Engineering including Data Modeling, Infrastructure setup on cloud, Data Warehousing and Data Lake development.
User: san089
data data-engineering data-engineering-pipeline etl-pipeline cassandra-database postgresql-database data-modeling data-warehouse data-lake airflow

teamclairvoyant / airflow-maintenance-dags
airflow,A series of DAGs/Workflows to help maintain the operation of Airflow
Organization: teamclairvoyant
airflow dag workflow airflow-maintenance-dags maintenance cleanup apache-airflow

teamclairvoyant / airflow-rest-api-plugin
airflow,A plugin for Apache Airflow that exposes rest end points for the Command Line Interfaces
Organization: teamclairvoyant
airflow rest-api plugin apache-airflow airflow-webserver airflow-plugin

teamclairvoyant / airflow-scheduler-failover-controller
airflow,A process that runs in unison with Apache Airflow to control the Scheduler process to ensure High Availability
Organization: teamclairvoyant
airflow scheduler-failover-controller apache-airflow airflow-scheduler airflow-cluster high-availability

timkpaine / paperboy
airflow,A web frontend for scheduling Jupyter notebook reports
User: timkpaine
jupyter jupyterlab jupyter-notebooks papermill nteract airflow apache-airflow phosphorjs dask scheduling-notebooks

villasv / aws-airflow-stack
airflow,Turbine: the bare metals that gets you Airflow
User: villasv
Home Page: https://victor.villas/aws-airflow-stack/
airflow airflow-cluster airflow-cookbook aws aws-cloudformation

webankfintech / dataspherestudio
airflow,DataSphereStudio is a one stop data application development& management portal, covering scenarios including data exchange, desensitization/cleansing, analysis/mining, quality measurement, visualization, and task scheduling.
Organization: webankfintech
Home Page: https://github.com/WeBankFinTech/DataSphereStudio-Doc
workflow governance azkaban davinci linkis spark hive hadoop visualis zeppelin

weibocom / rill-flow
airflow, Rill Flow is a high-performance, scalable distributed workflow orchestration service
Organization: weibocom
Home Page: https://rill-flow.github.io
airflow cloud-native dag job-scheduler orchestration task-scheduler workflow