This repository is the final project for Data Engineering Zoomcamp hosted by our good friends DataTalks Club. In this project, I selected the COVID-19 Cases in Toronto Data from Toronto Open Data Catalogue. The criteria for my selection of dataset is that it refreshed in a specific period or at least on a schedule. In this case, the dataset refreshes weekly. I am also interested in the COVID-19 cases in Toronto because some of my friends and family members live there.
The final output is a dashboard that shows some summary statistics of the COVID-19 cases in Toronto. The dashboard is deployed on Looker Studio. You can access the dashboard here
This project utilizes the following tech stack:
- Python script and Prefect for data pipeline and orchestration
- Google Cloud Platform
- Google Cloud Storage was used to store the raw data
- Google BigQuery was used to store the transformed data
- Google Looker Studio was used to create the dashboard
- Terraform for infrastructure as code
- dbt for more data transformation
The data pipeline consists of the following steps:
- Use Terraform to create a Google Cloud Storage bucket and a Google BigQuery dataset
- Extract the data from the Toronto Open Data Catalogue using their API
- Perform preliminary data transformation using Python script
- Store the raw data in Google Cloud Storage
- From the raw data, create a new table in Google BigQuery
- Perform more data transformation using dbt
- Create a dashboard using Google Looker Studio
Note: These set of steps assume that the user already created necessary Google Cloud Platform steps such as creating Prefect blocks and storing the credentials in a JSON file. The user should also have Terraform installed on their machine.
- Clone this repository
git clone [email protected]:jplaulau14/covid-19-toronto.git
- Create a virtual environment
python3 -m venv venv
- Activate the virtual environment For Mac/Linux
source venv/bin/activate
For Windows
venv\Scripts\activate
- Install the requirements
pip install -r requirements.txt
- Run the Bash Script for Terraform
chmod +x scripts/terraform.sh
./scripts/terraform.sh
- Run the Python Script
python flows/main.py
- Create a CI/CD pipeline to automate the deployment of the data pipeline
- Use Prefect Cloud to schedule weekly processing
- Add a detailed instruction on running this project locally