alexxnica / distributary

A data-flow based database with automatic materialization

distributary: a data-flow based database with automatic materialization

This repository provides an implementation of the data storage system model proposed in Soup.

At a high level, it takes a set of parameterized queries, and produces a data-flow graph that maintains materialized views for the output of those queries. This yields very high read throughput. Incremental maintenance of the views through the data-flow graph also yields high write throughput.

Code

The code is written in Rust, and its main entrypoint can be found in src/lib.rs. That file also contains a more extensive description of the internal system architecture.

Building

You need nightly Rust to run this code. rustup.rs is the recommended tool for maintaining multiple versions of Rust (or even for keeping a single nightly install up to date).

Once you have nightly rust, you can build the library

$ cargo build

Run the test suite

$ cargo test

Build the documentation (in target/doc/distributary/)

$ cargo doc

And run the example

$ cargo run --example basic-recipe

Benchmarks

Soup comes with a relatively simple benchmarking tool that executes reads and writes in a schema with articles and votes. Each write is a vote for a single article (all articles are pre-populated), and the reads are for the title and vote count for random articles.

To build the benchmarking tool, run

$ cargo build --bin vote --release

The --release enables compilation optimizations, and is necessary to get sensible numbers. Note that compiling with optimizations will also take longer than a regular debug build.

Once compiled, the benchmarker can be run with

$ target/release/vote --runtime 30 --articles 10000 soup://

Benchmarks against other targets

To compare distributary's performance against other backends, the vote binary has to be compiled with the appropriate feature enabled. You can enable multiple features by compiling with --features "b_x b_y". The additional features pull in more dependencies, so compilation will be slower.

target	--features	invocation argument	notes
Embedded soup	none	soup://
Soup over loopback	b_netsoup	netsoup://127.0.0.1:7777
MySQL	b_mysql	mysql://[email protected]/bench_soup	soup is the user, bench_soup the database -- note that the user must be allowed to drop and re-create the database.
SQL Server	b_mssql	mssql://server=tcp:127.0.0.1,1433;user=soup;pwd=password;/bench_soup	soup is the user, password their password, bench_soup the database -- note that the user must be allowed to drop and re-create the database.
PostgreSQL	b_postgresql	postgresql://[email protected]/bench_soup	soup is the user, bench_soup the database -- note that the user must be allowed to drop and re-create the database.
memcached	b_memcached	memcached://127.0.0.1:11211	the benchmarker does not purge the cache before starting, so you may want to restart memcached before running.