API for tracking cluster membership about nuraft

[kishorekrd]

Hi @greensky00 ,
Thanks for your responses. Any suggestions on the API for cluster membership tracking?
Thanks

[greensky00]

Hi @kishorekrd

"membership" itself never changes according to online/offline status, unless users explicitly do add/remove server to/from the cluster.

The term "offline" is ambiguous and cannot be defined clearly due to millions of situations as follows:

• The follower is still "online", but the leader just cannot see it due to network partition.
• The follower is still "online", but just slow so that cannot respond to a few heartbeats.
• The follower is still "online", and still responds to heartbeats, but its log index is lagging behind the majority of members. And the log gap is getting bigger.
• The leader is isolated so that it thinks all the others are "offline", but actually the other members are alive; they form a quorum and elect a new leader, but the old leader does not know about that. From the other members' point of view, the old leader is "offline".

Hence, keeping tracking of "offline" status cannot give us meaningful information for the Raft protocol itself. If you want to track "offline" status, you need to define "what offline is", and implement it outside Raft.

Please note that we provide a few callbacks for the "freshness" of followers:

NuRaft/include/libnuraft/callback.hxx

Lines 99 to 113 in 8699f3b

	/**
	* The difference of committed log index between the follower and the
	* master became smaller than a user-specified threshold.
	* Happens on follower only.
	* ctx: null.
	*/
	BecomeFresh = 12,
	/**
	* The difference of committed log index between the follower and the
	* master became larger than a user-specified threshold.
	* Happens on follwer only.
	* ctx: null
	*/
	BecomeStale = 13,

[greensky00]

+)
One thing NuRaft can do additionally is providing a new API that reports "the last time each follower responded", like the below:

follower1: responded 10 ms ago
follower2: responded 900 ms ago
follower3: responded 5,120 ms ago

From that information, you can decide the up/down status of each follower based on your criteria.

Please let me know if it works for you. Thanks.

[kishorekrd]

Thanks for the response. What is the callback function for the leader to call if any follower is not responding or some follower, which went down before came back ?

[greensky00]

@kishorekrd
There is no such callback as response failure is quite common although followers are alive. It will be super verbose and will cause lots of false positives.

[kishorekrd]

That make sense. Thanks.
in some thread you mentioned that , auto_forward is not used and not tested fully. Is that means clients has to send their requests to the leader always? Like 2 step method, connect to any node and find the leader details, then send request to the leader.

[greensky00]

connect to any node and find the leader details, then send request to the leader.

Yes. But usually, a leader does not change frequently so that you can remember the last leader and directly reach out to the leader next time. Then the 2-step method will happen only when the leader changes.

What about the API that I mentioned here? Do you think it will be helpful for you?
#199 (comment)

[kishorekrd]

Yes, API that you mentioned at #199 (comment) is helpful. Is this API already available or planing to add?

[greensky00]

Ok. It is a brand new API and will be added soon.

[kishorekrd]

Thanks,