savadev / spark-movies-etl

Spark data pipeline that ingests and transforms movie ratings data.

Movies data ETL (Spark)

Spark data pipeline that ingests and transforms movie ratings data.

Data Architecture

We define a Data Lakehouse architecture with the following layers:

• Raw: Contains raw data files directly ingested from an event stream, e.g. Kafka. Data is not catalogued and should generally not be accessible (can contain PII).
• Standardized: Contains standardized data (catalogued tables) based on the raw data but without any transformations applied (besides masking of PII data if necessary).
• Curated: Contains transformed data (catalogued tables) according to business and data quality rules.

Apache Iceberg is used as the table format.

Data pipeline design

The data pipeline consists of the following tasks:

• Standardize task: ingests the dataset from the Raw layer into the Standardized one.
• Curate task: consumes the dataset from Standardized, performs transformations and business logic, and persists into Curated.

The datasets are partitioned by execution date.

Execution instructions

The repo includes a Makefile. Please run make help to see usage.

Configuration management

Configuration is managed by the ConfigManager class, which is a wrapper around Dynaconf.

Packaging and dependency management

Poetry is used for Python packaging and dependency management.

CI/CD

Github Actions workflows for CI/CD are defined here and can be seen here.

The logic is as follows:

• On PR creation/update:
  • Run code checks and tests.
  • Build app (*).
  • Release to S3 (to a specific location for the PR, e.g. s3://movies-binaries/movies-etl/PR-123).
• On push to master:
  • Run code checks and tests.
  • Build app (*).
  • Release to S3 (to the location for the master version, e.g. s3://movies-binaries/movies-etl/latest).
  • Create Github release.

(*) The app build contains:

• The Python entrypoint file.
• A zip containing all the dependencies (Python packages).
• The app_config.yaml file.

Orchestration

An example Airflow DAG to run this pipeline on a schedule can be found here.