Question about auto_adjust_quorum_for_small_cluster_ option about nuraft

[greensky00]

Hi @sheepgrass

First of all, please note that you will take the risk of potential data loss if you use that option.

Your concern is valid, and if there are two leaders, we deterministically choose one of them as the actual leader and the other one as a follower. We are doing it automatically by defining a custom term counter here:

NuRaft/include/libnuraft/raft_server.hxx

Lines 578 to 581 in 2eb5e54

	/**
	* Set a custom callback function for increasing term.
	*/
	void set_inc_term_func(srv_state::inc_term_func func);

In the original Raft, the term increases monotonically: 1, 2, 3, 4, ... so that term may duplicate if there are two leaders, which causes data diverging.

Instead, you can make the term of each server unique if you define the term counter like this:

new_term = (int)((prev_term + 10) / 10) * 10 + (server's unique number)

For example, let's suppose the current term is 13, and there are two servers: S1 and S2. And you prefer S1 if two leader situation happens. So let's give the unique number of S1 and S2 as 5 and 3, respectively.

Once network partition happens, S1 and S2 become leader separately, then their new term will be

S1: (int)((13+10) / 10) * 10 + 5 = 25
S2: (int)((13+10) / 10) * 10 + 3 = 23

After the partition is gone, they will see each other, and then S1 will likely take the leadership due to (relatively) the higher term. rollback will be invoked on S2's side until it meets the log with the common term between the two, and then S2 will do catch-up with S1's data.

[sheepgrass]

Hi @greensky00

Thanks for your explanation of how to handle data diverging issue. I would rather not to use this option as data diverging/having two leaders is not acceptable in my application. Thanks~