ashwin153 / beaker

A distributed, transactional key-value store.

Beaker is a distributed, transactional key-value store. Beaker is N/2 fault tolerant but assumes that failures are fail-stop and that messages are received in order. Beaker is strongly consistent; conflicting transactions are linearized. Beaker features monotonic reads, read-your-write consistency, and tunable availability and partition tolerance.

Structure

# Database
beaker/                             https://github.com/ashwin153/beaker
+---beaker-assets/                  Documentation, results, and graphics.
+---beaker-benchmark/               Performance tests.
+---beaker-client/                  Client library.
+---beaker-common/                  Tools.
+---beaker-server/                  Database server.
+---build-support/                  Pants plugins and configuration.

Requirements

• Java 1.8
• Python 2.7 (build-only)
• Scala 2.12

beaker's Issues

Partial Scan

Allow ranges of keys to be specified to the scan method to the Client that allows for ranges of keys to be scanned. Right now we may only scan the entire key-space.

Tunable Consistency

Use Ballot instead of version in Revisions. In a consistent system, they are equivalent. However, if we use Ballots instead of versions we could tune the consistency of the system as well. This would allow different partitions to simultaneously commit conflicting transactions and repair them when they reconnect, because each partition will write different versions of the key.

RocksDB Integration

Memcached Integration

TLA Spec for Consensus

Benchmarking

Performance testing is lacking. Integrate with YCSB, TCP-C/E, etc.

io.grpc.StatusRuntimeException: CANCELLED: call already cancelled

Likely due to scalapb/ScalaPB#438.

Consensus Memory Footprint

Right now the Paxos state for a transactions cannot be cleared until a newer, conflicting transaction is proposed. For example, an accepted transaction will continue to be stored in memory until another conflicting transaction is accepted. Over time, the memory footprint of the program could become massive.

At this point in time, I'm not sure of an elegant solution to this problem. One solution might be to introduce timeouts, after which the Paxos state for a transaction is automatically cleared. However, synchronizing timeouts across the cluster is a consensus problem in and of itself.

LevelDB Integration

Cassandra Integration

Rust Rewrite

Rust's performance is on par with C, and its expressiveness and elegance are on par with Scala. Unfortunately, it doesn't look like Pants supports Rust yet (even though its engine is written in Rust?!). However, there are Bazel rules for both Scala and Rust. Switching to Bazel shouldn't be too challenging, and might even improve build times.

Instance Configuration

Use PureConfig to bootstrap Instances from a configuration file.

Clear Linux Support

Benchmarks and features make it sound promising for our workloads. AWS runs exclusively on Intel hardware, so we'll be able to make use of the CPU-specific optimizations in Clear Linux. We'll need to verify that Pants supports Clear Linux.

Cap'n Proto

GRPC is over HTTP/2, while Cap'n Proto is over TCP. This should significantly improve network performance.

Logging

Right now there is no visibility into server instances. Only way to debug is to add print statements and run in a REPL.

Protobuf Migration

Cluster Reconfiguration

Multithreading Race Condition

Beakers are vulnerable to race conditions because they use default collections. Therefore, concurrent requests on a beaker can lead to indeterminate behavior. Instead, use the equivalent concurrent collections to store state.

Redis Integration

Performance

• Lower consensus backoff duration to slightly more than the 99% latency.
• Fail-fast if a proposal conflicts with a proposal on the same beaker.
• SortedMap to ConcurrentSkipListMap.
• Network and memory optimized EC2 instances.
• Increase JVM heap size to reduce GC pressure.
• EC2 Placement Groups to reduce latency.
• Do not perform consensus if there is only one beaker in the cluster.
• Allow prepare and learn to occur simultaneously.
• Executor look for latest conflicting transaction from right-to-left, not left-to-right.
• Return if a majority accept the proposal; don't wait for the proposal to be learned.

Tail Recursive Consensus

Consensus is already tail recursive; however, the Scala compiler isn't smart enough to know it. If we replace the various flatMap calls with match expressions, the Scala compiler will recognize the function as tail-call optimizable.