A Rust implementation of a distributed replication protocol providing:
- Strong consistency (linearizability)
- High availability through replication
- Low latency local reads
- Decentralized peer-to-peer architecture
use hermes_replication::ClusterNode;
use std::net::SocketAddr;
// Create a node
let node = ClusterNode::new(
"127.0.0.1:8080".parse().unwrap()
);
// Write and read data
async {
node.write(b"key".to_vec(), b"value".to_vec()).await?;
let value = node.read(b"key".to_vec()).await?;
assert_eq!(value, b"value");
};-
Coordinator Selection:
- Each key maps to a specific coordinator node
- Mapping uses consistent hashing of the key
- Any node can be coordinator for different keys
-
Write operation flow:
- Node checks if it's coordinator for the key
- Coordinator sends validation to all active replicas
- Updates local copy after successful validation
- Write completes when all replicas acknowledge
- Any active node can serve reads if:
- Node has valid local copy
- Node is in active replica set
- If local copy invalid:
- Fetch from other replicas
- Update local copy
- Serve read
- Globally unique versions using node ID and counter
- High 16 bits: Node identifier
- Low 48 bits: Monotonic counter
- Ensures consistent ordering of concurrent writes
- NotResponsible: Node is not coordinator for key
- NoActiveReplicas: No active nodes available
- NetworkError: Communication failure
- QuorumFailed: Failed to reach required replicas
- ValueNotFound: Key does not exist
- Active: Node can participate in operations
- Inactive: Node cannot participate in operations
- Suspected: Node is potentially unhealthy
| # | Test Case | Description | Expected Result |
|---|---|---|---|
| Cluster Tests | |||
| 1 | test_three_node_cluster | Initialize a 3-node cluster and verify basic network connectivity and membership protocols | • Successful node joins • Heartbeat responses • Complete membership list |
| 2 | test_write_and_read | Perform coordinated write operation and verify data consistency across all nodes | • Write completes successfully • All nodes have identical data • Correct version numbers |
| 3 | test_concurrent_writes | Execute multiple simultaneous writes to the same key to test conflict resolution | • All nodes converge to same value • Consistent version ordering • No data corruption |
| Hermes Protocol Tests | |||
| 4.1 | Coordinator Selection | Verify that only the designated coordinator can write to a key | • Write from non-coordinator fails with NotResponsible • Write from coordinator succeeds |
| 4.2 | Write Protocol & Versioning | Test write replication and version synchronization | • All nodes receive the write • All nodes have identical version numbers • Version numbers match across replicas |
| 4.3 | Local vs Forwarded Reads | Compare reads from coordinator and non-coordinator nodes | • Both reads return same value • Coordinator reads succeed • Non-coordinator reads succeed |
| 4.4 | Version Increment | Verify version numbers increment properly on updates | • New version > old version • All nodes update to new version • Data consistency maintained |
| 4.5 | Multiple Key Operations | Test operations on multiple keys with different coordinators | • Each key maintains correct value • Different coordinators can operate independently • No cross-key interference |
| Failure Tests | |||
| 5 | test_write_failures | Test write operation failure modes: • Write to wrong coordinator • Write to inactive cluster |
• NotResponsible error when wrong coordinator • NoActiveReplicas error when cluster inactive |
| 6 | test_read_failures | Test read operation failure modes: • Read non-existent data • Read from inactive node |
• ValueNotFound error for missing data • NoActiveReplicas error for inactive node |
| 7 | test_network_failures | Simulate network partition by disconnecting nodes during operation | • QuorumFailed error when unable to reach sufficient replicas • Consistent state after recovery |
| 8 | test_quorum_failures | Test behavior when insufficient nodes are available for consensus | • QuorumFailed error when consensus impossible • No partial updates |
- Multi-node cluster setup
- Write/read operations
- Node state changes
- Membership changes
- Concurrent write handling
- Write coordination failures
- Read failures
- Network failures
- Quorum failures
✅ Basic replication protocol
✅ Key-based coordinator selection
✅ Local reads from valid copies
✅ Error handling
✅ Network communication
✅ Test coverage
✅ Concurrent write handling
✅ Version management
⏳ Persistence (TODO)
⏳ Node recovery (TODO)
⏳ Consistent hashing for replica selection (TODO)
⏳ Full invalidation phase (TODO)
