This is an ETL (Extract, Transform, Load) project that uses the Chinook database as the data source.
The Chinook database is a sample database that represents a digital media store. The database contains information about the artists, songs, and albums from the music store, as well as information on the store's employees, customers, and the customers purchases.
The goal of this project is to create an ETL pipeline that will extract data from the Chinook database, transform the data, and load the data into a Google Cloud Platform (GCP) BigQuery data warehouse. The data warehouse will be used to analyze the data and answer business questions.
-
Go to
data/
directory. -
Login to your local PostgreSQL.
-
Dump the Chinook SQL schema by running:
\i chinook_pg.sql
-
To check if the database is created, run the following command:
SELECT tablename FROM pg_catalog.pg_tables WHERE schemaname != 'pg_catalog' AND schemaname != 'information_schema';
Ensure there are 11 tables listed.
Coming Soon...
-
Set up virtual environment and install dependencies.
python3 -m venv venv source venv/bin/activate
-
Setup environment variables.
echo -e "AIRFLOW_UID=$(id -u)\nAIRFLOW_GID=0" > .env
-
Add Airflow's internal Postgres and GCP resources to the
.env
. Follow.env.example
for reference. -
Run docker container for Airflow.
docker-compose up --build -d
-
Go to Airflow Web UI at http://localhost:8080/ to ensure the container is running.
To connect Airflow with external system, we need to setup connections on Airflow.
-
On http://localhost:8080, go to Admin > Connections. Add Google Cloud connection type.
- Conn Id:
google_cloud_conn_id
- Conn Type:
Google Cloud Platform
- Project Id:
<your-project-id>
- Keyfile Path:
/creds/<your-sa-keyfile>.json
- Scopes:
https://www.googleapis.com/auth/cloud-platform
- Conn Id:
-
Add another connection for the local PostgreSQL.
- Conn Id:
postgres_conn_id
- Conn Type:
Postgres
- Host:
<your-local-ip>
(e.g.: 192.x.x.x) - Schema:
<your-local-postgres-schema>
- Login:
<your-local-postgres-username>
- Password:
<your-local-postgres-password>
- Port:
5432
- Conn Id:
Go to http://localhost:8080 to turn on the DAG.
The DAG scheduled to run @monthly
from inclusive range of 2009-01-01T05:00
to 2013-12-01T05:00
.
-
ETL Flatfile Airflow by @okzapradhana
-
Chinook Database by @lerocha