eljandoubi / aws-data-warehouse

Build an ETL pipeline for a database hosted on AWS Redshift.

Data-Warehouse

Overview

This project involves building an ETL pipeline for a music streaming startup. The datasets provided include song data and log data, both stored in Amazon S3 buckets. The song dataset is a subset of the Million Song Dataset, containing JSON files with metadata about songs and their artists. The log dataset consists of JSON files generated by an event simulator, simulating app activity logs from the music streaming app.

Dataset Description

Song Dataset

The song dataset contains JSON files partitioned by the first three letters of each song's track ID. Each file provides metadata about a song and its corresponding artist.

Log Dataset

The log dataset comprises JSON files partitioned by year and month, simulating app activity logs from the music streaming app. These logs include user interactions such as song plays.

Schema

The project employs a star schema with one fact table and multiple dimension tables:

Fact Table

• songplays: Records in log data associated with song plays (i.e., records with the page NextSong). Columns include songplay_id, start_time, user_id, level, song_id, artist_id, session_id, location, and user_agent.

Dimension Tables

1. users: Users in the app. Columns include user_id, first_name, last_name, gender, and level.
2. songs: Songs in the music database. Columns include song_id, title, artist_id, year, and duration.
3. artists: Artists in the music database. Columns include artist_id, name, location, latitude, and longitude.
4. time: Timestamps of records in songplays broken down into specific units. Columns include start_time, hour, day, week, month, year, and weekday.

Project Structure

Code Files

1. create_table.py: Creates fact and dimension tables as well as staging tables for the star schema in Redshift.
2. etl.py: Loads data from S3 into staging tables on Redshift and then processes that data into analytics tables on Redshift.
3. sql_queries.py: Defines SQL statements imported into other files.
4. test.ipynb: Creates a Redshift cluster, IAM role, and verifies the result after running etl.py.
5. README.md: Documentation providing insights into the project process and decisions for the ETL pipeline.

Project Steps

Table Schemas Design

1. Design schemas for fact and dimension tables.
2. Write SQL CREATE statements for each table in sql_queries.py.
3. Complete logic in create_tables.py to connect to the database and create tables.
4. Write SQL DROP statements in create_tables.py to drop tables if they already exist, enabling resetting of the database for pipeline testing.
5. Launch a Redshift cluster and create an IAM role with S3 read access.
6. Add Redshift database and IAM role info to dwh.cfg.
7. Test by running create_tables.py and checking table schemas in the Redshift database.

ETL Pipeline Implementation

1. Implement logic in etl.py to load data from S3 to staging tables on Redshift.
2. Implement logic in etl.py to load data from staging tables to analytics tables on Redshift.
3. Test by running etl.py after running create_tables.py and running analytic queries on Redshift database to compare results with expected results.
4. Delete the Redshift cluster when finished.

Running Scripts

1. Set environment variables KEY and SECRET.
2. Choose DB/DB_PASSWORD in dhw.cfg.
3. Create IAM role, Redshift cluster, connect to S3 bucket, and configure TCP connectivity.
4. Drop and recreate tables:

   $ python create_tables.py

5. Run the ETL pipeline:

   $ python etl.py

6. Delete IAM role and Redshift cluster.