unmeshjoshi / replicate

Replicate

Note: This framework was built while working on the book Patterns of Distributed Systems. I use this to teach replication techniques in the workshops that I conduct.

/*                                 message1     +--------+
                                +-------------->+        |
                                |               | node2  |
                                |    +-message2-+        |
                                |    |          |        |
                             +--+----v+         +--------+
 +------+   request-response |        |
 |      |                    |node1   |
 |client| <--------------->  |        |
 |      |                    |        |
 +------+                    +-+----+-+
                               |    ^            +---------+
                               |    |            |         |
                               |    +--message4--+ node3   |
                               |                 |         |
                               +--message3------->         |
                                                 +---------+
*/                              

Overview

This is a basic framework for quickly building and testing replication algorithms. It doesn't require any additional setup (e.g., Docker) for setting up a cluster and allows writing simple JUnit tests for testing replication mechanisms. The framework was created to learn and teach various distributed system techniques, enabling the testing of working code while exploring distributed systems concepts. It provides mechanisms for establishing message-passing communication between replicas and test utilities for quickly forming a cluster of replicas, introducing network failures, and asserting the state of the replicas. This repository also contains example code for basic replication algorithms like basic Majority Quorum, Paxos, MultiPaxos and Viewstamped Replication.

Basic Design

This framework allows you to implement replication algorithms quickly and write JUnit tests. It also offers basic ways to introduce faults like process crashes, network disconnections, network delays, and clock skews.

Replica Class

The Replica class implements essential building blocks for a networked service, including:

• Listening on provided IP addresses and ports.
• Managing a single-threaded executor for request handling, Implementing the Singular Update Queue pattern.
• Providing a basic implementation of the Request Waiting List pattern.
• Supporting serialization and deserialization of messages (currently using JSON).

These building blocks are sufficient for implementing and testing any networked service.

Writing JUnit Tests

Utilities are provided to create multiple Replica instances. These instances, being Java objects, are easy to inspect and test. Check out QuorumKVStoreTest for an example of how to write tests. The cluster can be formed by creating multiple instances of the Replica implementations what you create. For example, a three node cluster with replicas named "athens", "byzantium" and "cyrene" is created as following:

class QuorumKVStoreTest {
  QuorumKVStore athens;
  QuorumKVStore byzantium;
  QuorumKVStore cyrene;
  
  @Override
  public void setUp() throws IOException {
    //no. servers = no. of replicas.
    this.nodes = TestUtils.startCluster(Arrays.asList("athens",
                    "byzantium", "cyrene"),
            (name, config, clock, clientConnectionAddress, peerConnectionAddress, peerAddresses) -> new QuorumKVStore(name, config, clock, clientConnectionAddress, peerConnectionAddress, peerAddresses));

    athens = nodes.get("athens");
    byzantium = nodes.get("byzantium");
    cyrene = nodes.get("cyrene");
  }
}

Introducing Failures

The Replica class allows you to introduce network failures to other nodes, with utility methods for dropping or delaying messages. Examples of testing with introduced network failures can be found in QuorumKVStoreTest. For example, to drop messages from the node 'athens' to 'byzantium'

 athens.dropMessagesTo(byzantium);

The connection can be restablished using

 athens.reconnectTo(cyrene);

Available Replication Algorithms

This repository contains example replication mechanisms, including:

1. Basic Read/Write Quorum
2. Quorum Consensus
3. Single-value Paxos
4. Paxos-based Key-Value Store
5. Paxos-based Replicated Log
6. MultiPaxos
7. View Stamped Replication

Explore these algorithms to understand and experiment with different replication techniques.

replicate's Issues

Client blocks if no readTimeout is set

replicator/src/main/java/replicate/net/SocketClient.java blocks in blockingSendAndReceive if no read timeout is set.
Ideally, the server should return error response in case request futures in the async processing pipeline do not complete.

Conflict in random ports assigned in tests

The tests are run by assigning random ports in method TestUtils.getRandomPort. It looks like the ports conflict once in a while.
Kafka had similar issue https://issues.apache.org/jira/browse/KAFKA-1501 and they discourage use of this method for assigning random ports in tests.

There is no entry point for this code

There is no main method. How is this code intended to be executed? Only through unit tests?

Also, the Docker file mentions Go. What is Go used for in this project?