Spark data pipeline that ingests and transforms movie ratings data.
We define a Data Lakehouse architecture with the following layers:
Raw
: Contains raw data files directly ingested from an event stream, e.g. Kafka. Data is not catalogued and should generally not be accessible (can contain PII).Standardized
: Contains standardized data (catalogued tables) based on the raw data but without any transformations applied (besides masking of PII data if necessary).Curated
: Contains transformed data (catalogued tables) according to business and data quality rules.
Apache Iceberg is used as the table format.
The data pipeline consists of the following tasks:
- Standardize task: ingests the dataset from the
Raw
layer into theStandardized
one. - Curate task: consumes the dataset from
Standardized
, performs transformations and business logic, and persists intoCurated
.
The datasets are partitioned by execution date.
The repo includes a Makefile
. Please run make help
to see usage.
Configuration is managed by the ConfigManager class, which is a wrapper around Dynaconf.
Poetry is used for Python packaging and dependency management.
Github Actions workflows for CI/CD are defined here and can be seen here.
The logic is as follows:
- On PR creation/update:
- Run code checks and tests.
- Build app (*).
- Release to S3 (to a specific location for the PR, e.g.
s3://movies-binaries/movies-etl/PR-123
).
- On push to master:
- Run code checks and tests.
- Build app (*).
- Release to S3 (to the location for the master version, e.g.
s3://movies-binaries/movies-etl/latest
). - Create Github release.
(*) The app build contains:
- The Python entrypoint file.
- A zip containing all the dependencies (Python packages).
- The
app_config.yaml
file.
An example Airflow DAG to run this pipeline on a schedule can be found here.