sgouda0412 Goto Github PK

etl-cloud-batch-processing

Set up a Google Cloud Composer environment and solve several batch-processing cases by creating DAGs to run ETL jobs in the cloud

etl-flatfile-airflow

Building Data Warehouse on BigQuery which takes flat file as the data sources with Airflow as the Orchestrator

etl-pipeline

Educational project on how to build an ETL (Extract, Transform, Load) data pipeline, orchestrated with Airflow.

etl-pipeline--orchestration-with-apache-airflow

This GitHub repository contains an Apache Airflow DAG script for orchestrating AWS Glue ETL jobs. It automates the execution of Glue jobs , with defined dependencies and waiting periods. Also includes setting up and how to use DAG on Apache Airflow

etl-pipeline-example

An example of an ETL pipeline that lays out generic DE processes. This is now out of date but still provides useful information

etl-pipeline-from-scratch

An ETL pipeline to efficiently extract, transform and load data into PostgresSQL table

etl-pipeline-glue_s3_redshift_bi

Building Data Pipeline using Apache Airflow, Glue, S3, Redshift, PowerBI . Performing Customer Churn Data Analytics

etl-pipeline-using-airflow

ETL pipeline to extract data from AWS S3 and transform it and load it to AWS Redshift

etl-with-apache-airflow

ETL Process with Apache Airflow fetching data from Ministerio de Salud de la Nación Argentina

etl_import_financial_data_to_postgresql

Simple ETL script with Apache Airflow , downloading financial data from alphavantage trought API , trasform into pandas dataframe and uplod to PostgreSQL

etl_pipelines

etl_processing

I find Apache Airflow very useful for ETL work. Here I transferred data from the source database(mysql) to the target database(postresql) and used the Airflow Bash Operator.

etl_project

Extracting covid data from API, Transforming the data and load the data on to sql table ( in Mysql on Aws). Also, setting up Cron job on Linux server inside AWS(EC2).

etlshopee_menshoes

evaluation_project

every-programmer-should-know

A collection of (mostly) technical things every software developer should know about

exchange-rate-prediction

fastapi-docker-kubernetes-streamlit

fastapi-observability

Observe FastAPI app with three pillars of observability: Traces (Tempo), Metrics (Prometheus), Logs (Loki) on Grafana through OpenTelemetry and OpenMetrics.

file-transfer-ftp-to-s3-python

Python 3 script to transfer files from FTP server to AWS S3 bucket

final_demo

this is for liveability final demo

football-data-scrapper-etl-to-postgresql

In this project, I build a web scrapper to scrappe: football-data.co.uk website and performed Extraction Transformation & loading (ETL) on the screapped data to a Mysql database

gcp-mlops-airflow

gcp-utils

gcp_dr_automation

gcp_sendgrid_smtp_config

DEMO: Google Cloud Composer (Airflow v2) SMTP Email Configuration

gcpbucket-to-bigquery-airflow

Using Airflow in Google Composer to orchestrate files loading from GCP Bucket to a partitioned table in BigQuery

gh-traffic

Using GH Traffic Data to learn about our users.

git-test

Learning git repos

git_101

Brief introduction to Git. List of some of the basic version control commands.