antonlevashov / ny_taxi

Syllabus

Syllabus

Note: NYC TLC changed the format of the data we use to parquet. But you can still access the csv files here.

Week 1: Introduction & Prerequisites

• Course overview
• Introduction to GCP
• Docker and docker-compose
• Running Postgres locally with Docker
• Setting up infrastructure on GCP with Terraform
• Preparing the environment for the course

Week 2: Workflow Orchestration

• Data Lake
• Workflow orchestration
• Introduction to Prefect
• ETL with GCP & Prefect
• Parametrizing workflows
• Prefect Cloud and additional resources

Week 3: Data Warehouse

• Data Warehouse
• BigQuery
• Partitioning and clustering
• BigQuery best practices
• Internals of BigQuery
• Integrating BigQuery with Airflow
• BigQuery Machine Learning

Week 4: Analytics engineering

• Basics of analytics engineering
• dbt (data build tool)
• BigQuery and dbt
• Postgres and dbt
• dbt models
• Testing and documenting
• Deployment to the cloud and locally
• Visualizing the data with google data studio and metabase

Week 5: Batch processing

• Batch processing
• What is Spark
• Spark Dataframes
• Spark SQL
• Internals: GroupBy and joins

Week 6: Streaming

• Introduction to Kafka
• Schemas (avro)
• Kafka Streams
• Kafka Connect and KSQL

Workshop: Maximizing Confidence in Your Data Model Changes with dbt and PipeRider

More details

Overview

Architecture diagram

Technologies

• Google Cloud Platform (GCP): Cloud-based auto-scaling platform by Google
  • Google Cloud Storage (GCS): Data Lake
  • BigQuery: Data Warehouse
• Terraform: Infrastructure-as-Code (IaC)
• Docker: Containerization
• SQL: Data Analysis & Exploration
• Prefect: Workflow Orchestration
• dbt: Data Transformation
• Spark: Distributed Processing
• Kafka: Streaming

Tools

For this course, you'll need to have the following software installed on your computer:

• Docker and Docker-Compose
• Python 3 (e.g. via Anaconda)
• Google Cloud SDK
• Terraform