Other project that uses NuRaft as a library about nuraft

[greensky00]

Hi @kishorekrd

Unfortunately, our other projects are not open-source now. But internals are pretty much similar to the example here
https://github.com/eBay/NuRaft/tree/master/examples/calculator

For the communication with clients, you need to replace CLI in this example with other protocols such as REST API or gRPC, and put the received requests to NuRaft.

[kishorekrd]

Hi @greensky00 ,
Thanks for your response. I have couple of more questions.
Does the leader send the heartbeat to learner servers?
If so, how is it scalable if there are large number of learner servers?
If not how the cluster membership is maintained for learner servers? If learner server goes down, how leader comes to know about it?
If a read request for the state machine comes to a Follower, does it read from its local copy or forwards the request to leader?
How Learner server handles the read requests for the state machine ?

Thanks for your help

[greensky00]

Hi @kishorekrd

Does the leader send the heartbeat to learner servers?

Yes.

how is it scalable if there are large number of learner servers?

Sending heartbeats (and other messages) are executed by Asio worker threads. It depends on the number of concurrent threads, the computing power of the machine, and the network capability. But anyway the net overhead will increase linearly according to the number of members (including learners) as the leader should send the same message to each member every time through TCP, so having 100-1000 learners won't work well.

does it read from its local copy or forwards the request to leader?

Read-path is not the part of Raft protocol, so it is up to you. In the projects with NuRaft in eBay, followers return local copy, and if a client wants to do a strongly-consistent read, it should read from the leader.

How Learner server handles the read requests for the state machine ?

In terms of state machine operations, there is no difference between learners and others. So it is the same as above: depending on how to implement the state machine. Learner is just a member who does not participate in the leader election.

[kishorekrd]

Hi @greensky00 ,
Thanks for your prompt responses. Couple of more questions.

What is the biggest single NuRaft cluster at ebay?
Based on your response, I see that 100-1000 node cluster is difficult. What do you do to scale beyond that?

I see some projects using gossip protocol for scaling beyond the raft cluster scale. Like small raft cluster for consistency and gossip protocol for the new nodes, which does not participate in the raft group, but learns its state machine with eventual consistency. What is your comment on that?

Thanks for your help

[greensky00]

Hi @kishorekrd

A single Raft cluster has less than 10 members in eBay. The size of a cluster defines the replication factor (as all members will have exactly the same data), not the horizontal scalability. To scale the system, we increase the number of Raft clusters, which share nothing among clusters. Quite similar to TiKV architecture that you can find here:
https://tikv.org/docs/3.0/concepts/architecture/

If what you meant was really scaling the Raft cluster itself (so that all members have the same data), one possible solution is having a learner in the cluster, and letting the learner broadcast the received data to other servers using streaming platforms like Kafka. Still, receivers' data will be eventually consistent though.

[kishorekrd]

Thanks a lot for the prompt response.

[zhanglistar]

does it read from its local copy or forwards the request to leader?

Read-path is not the part of Raft protocol, so it is up to you. In the projects with NuRaft in eBay, followers return local copy, and if a client wants to do a strongly-consistent read, it should read from the leader.

@greensky00 Can reading from leader achieve strongly-consistent in case of network partition occurs where original leader is not leader now?