ericbjames / northwindetl

NorthwindETL

Introduction: The purpose of this project is to optimize the operations of an e-commerce team by transforming raw data into a more organized and efficient format, and then presenting the findings to the management team. The pipeline follows a bronze-silver-gold architecture and uses Databricks for data processing. The final output of the pipeline is a set of tables and visualizations that provide valuable insights into the e-commerce operations of Northwind Traders.

Project Architecture

Data Source

The pipeline utilizes the Northwind sample database provided by Microsoft, which represents a fictional company called Northwind Traders.

Pipeline Stages

Bronze Stage

In this stage, raw data is ingested from AWS RDS using Airbyte and subsequently processed in Databricks. As the data is in JSON format, it must be parsed before being transformed into a structured format and stored in Delta tables. The following tables are selected for further processing:

customers orders orders_details

Silver Stage

During the silver stage, the data from the bronze tables is cleaned, transformed, and prepared for dimensional modeling. Data quality checks are performed using the Great Expectations library, which helps ensure the accuracy and consistency of the data.

Gold Stage

In the gold stage, advanced data processing and aggregation are performed to generate insights and facilitate reporting. The results can be visualized in dashboards, enabling management teams to make data-driven decisions.

Dimensional Model:

Data Quality and Validation

Data quality testing is performed using the Great Expectations library, which helps maintain the integrity of the data throughout the pipeline. Expectations are defined for each DataFrame, and the data is validated against these expectations to ensure its quality.

Databricks Workflow:

Final Dashboard

The final dashboard has the following visualizations:

• Total Revenue by Month (line chart)
• Top 10 Customers by Revenue (bar chart)
• AVG Shipping cost by Shipping Company (3 line chart)
• AVG Shipping cost by Country (map)