jazz-min / de_eventsim

Data Engineering End to end project - data pipeline for streaming data

de_eventsim

Data Engineering End to end project - data pipeline for streaming data with Kafka, Spark Streaming, dbt, Docker, Airflow, Terraform, GCP and much more!

Dataset

EventSim - Program that generates event data for a fake music web site (similar to Spotify).

Tools & Technologies

• Cloud - Google Cloud Platform

• Infrastructure as Code software - Terraform

• Containerization - Docker, Docker Compose

• Stream Processing - Kafka, Spark Streaming

• Orchestration - Airflow

• Transformation - dbt

• Data Lake - Google Cloud Storage

• Data Warehouse - BigQuery

• Data Visualization - Data Studio

• Language - Python

  Architecture

  --- Add image

  Final Result

  --- add dashboard image

Setup

Pre-requisites

• Google Cloud Platform.
  • GCP Account and Access Setup
• Terraform
  • Setup Terraform