sspaeti / datadex

Open Data Warehouse with dbt and DuckDB

D A T A D E X

Collaborate on Open Data using Open Source Tools

Open source, serverless, and local-first Data Platform to collaborate on Open Data! Built on top of Dagster, dbt, Quarto, DuckDB, and Evidence.

This repository is an up to date toy implementation of the overall pattern. You can check two real world production instances of Datadex working in the following repositories:

• Gitcoin Grants Data Portal. Improve data access and empower data scientists to conduct research and guide community driven analysis and decisions around Gitcoin Grants.
• Filecoin Data Portal. Local-first data hub for Filecoin data! Like Dune, but in your laptop.

💡 Principles

• Open: Code, standards, infrastructure, and data, are public and open source.
• Modular and Interoperable: Each component can be replaced, extended, or removed. Works well in many environments (your laptop, in a cluster, or from the browser), can be deployed to many places (S3 + GH Pages, IPFS, ...) and integrates with multiple tools (thanks to the Arrow ecosystem). Use open tool, standards, infrastructure, and share data in accesible formats.
• Permissionless. Don't ask, fork it and improve the models, add a new source or update any script. No API limits, just plain files.
• Data as Code. Declarative stateless transformations tracked in git. Version your data as code! Publish and share your reusable models for others to build on top. Datasets should be both reproducible and accessible!
• Glue: Be a bridge between tools and aproaches. E.g: Use software engineering good practices like types, tests, materialized views, and more.

Datadex is not a new tool. Datadex is a pattern showing an opinionated bridge between existing ones.

🚀 What can you do with Datadex?

• Model existing datasets using Python and SQL like you would do in your company. You can rely on awesome dbt features like tests and docs (automatically generated and published on GitHub Pages) too.
• Explore your data wherever you want. Use Jupyter Notebooks, BI Tools, Excel, .... Share your findings with others by publishing them online as beautiful static websites (thanks to Quarto).
• Add new data sources to the Datadex. The goal is simple; bring data locally and work with it!

⚙️ Setup

Datadex consists of several components and requires some setup to get started.

🐳 Docker / Dev Containers

The fastest way to start using Datadex is via VSCode Remote Containers. Once inside the develpment environment, you'll only need to run make dev to spin up the Dagster UI locally.

The development environment can also run in your browser thanks to GitHub Codespaces!

You can also build the Dockerfile image locally and run it with:

docker build -t datadex .
docker run -it -v $(pwd):/workspaces/datadex -p 3000:3000 datadex

🐍 Python Virtual Environment

Clone the repository and run the following commands from the root folder:

# Create a virtual environment
python3 -m venv .venv
source .venv/bin/activate

# Install the package and dependencies
pip install -e .[dev]

Now, you should be able to spin up Dagster UI and access it locally.

🎯 Motivation

This project started after thinking how an Open Data Protocol could look like!

👏 Acknowledgements

• This proof of concept was created thanks to open source projects like DuckDB, dbt, Dagster, and Quarto.
• Datadex name was inspired by Juan Benet's data projects.