Auto adding followers to leader about nuraft

[greensky00]

Hi @kishorekrd

First of all, once a cluster forms, a node is still a member of the cluster even though it goes down. After it comes back, it is not "re-added", but just the connection is recovered. Adding/removing a server is all about "cluster membership change", and irrelevant to "status of existing member". This is important because adding/removing a server will change the size of a quorum (i.e., the number of members to reach a consensus), while the online/offline status change of a member shouldn't affect it.

If what you want is "the list of servers successfully added to the cluster, so that will be a still member even after going down and up", there is an API for it:

NuRaft/include/libnuraft/raft_server.hxx

Lines 465 to 470 in 8a19e7d

	/**
	* Get the configuration of all servers.
	*
	* @param[out] configs_out Set of server configurations.
	*/
	void get_srv_config_all(std::vector< ptr<srv_config> >& configs_out) const;

It will return the list of members and their configurations.

[kishorekrd]

Hi @greensky00 ,

Thanks for your response. Yes, you are correct, if node goes down and comes back, it's connection is recovered. But if I try to add a down node with add_srv() API, it will fail and is not going to try again to add that node, even if it comes back up. Once the node added successfully, it does not matter whether node is up or down, get_srv_config_all() will show that node. Once it comes back it's connection is recovered. But if add_srv() API fails, its not going to try again. Is there any way to keep trying to add a node until it succeeds?

One way is to provide the list of remote node's details to each node and they all try to add all the remote nodes. Some will succeed and some will fail, but ultimately they form the cluster. There might be some race conditions here.

Is there any other way?

Thanks

[greensky00]

If you want to have a pre-defined set of servers at the beginning, the better way is to let state_mgr return clutser_config which contains the list of nodes:

NuRaft/include/libnuraft/state_mgr.hxx

Lines 35 to 46 in 8699f3b

	/**
	* Load the last saved cluster config.
	* This function will be invoked on initialization of
	* Raft server.
	*
	* Even at the very first initialization, it should
	* return proper initial cluster config, not `nullptr`.
	* The initial cluster config must include the server itself.
	*
	* @return Cluster config.
	*/
	virtual ptr<cluster_config> load_config() = 0;

then you don't need to add servers.

[kishorekrd]

I followed this suggestion and added the following code for load_config().
`
ptr<cluster_config> load_config() {
// Just return in-memory data in this example.
// May require reading from disk here, if it has been written to disk.

ptr<srv_config> peer_srv_config_;
for (int i = 0; i < num_servers_; i++) {     
  std::string peer_host(servers_[i]);
  std::string peer_endpoint = peer_host + std::string(":") + std::to_string(admin_ports_[i]);
  std::cout << " load_config server id " << i
            << "    endpoint:    " << peer_endpoint << std::endl;
  peer_srv_config_ = cs_new<srv_config>(i, peer_endpoint);
  saved_config_->get_servers().push_back(peer_srv_config_);
}
return saved_config_;

}
`
Here there are 3 servers. All are coming up at the same time with same cluster configuration.

but getting the following errors in the log and Raft init is failing. All failed with same issue. Am I missing anything?
init Raft instance . ................... FAILED

log file:
2021-04-16T23:04:57.837_356+00:00 [2c40] [====] Start logger: ../tmp/dx/2021-04-16_23-04-57.log (32 MB per file, up to 10 files) [logger.cc:928, start()] 2021-04-16T23:04:57.838_411+00:00 [2c40] [INFO] Raft ASIO listener initiated, UNSECURED [asio_service.cxx:706, asio_rpc_listener()] 2021-04-16T23:04:57.838_472+00:00 [2c40] [INFO] parameters: timeout 200 - 400, heartbeat 100, leadership expiry 2000, max batch 100, backoff 50, snapshot distance 5, log sync stop gap 99999, reserved logs 5, client timeout 3000, auto forwarding OFF, API call type BLOCKING, custom commit quorum size 0, custom election quorum size 0, snapshot receiver INCLUDED, leadership transfer wait time 0 [raft_server.cxx:367, apply_and_log_current_params()] 2021-04-16T23:04:57.838_488+00:00 [2c40] [INFO] new timeout range: 200 -- 400 [raft_server.cxx:317, update_rand_timeout()] 2021-04-16T23:04:57.838_507+00:00 [2c40] [INFO] === INIT RAFT SERVER === commit index 0 term 0 election timer allowed log store start 1, end 0 config log idx 0, prev log idx 0 [raft_server.cxx:149, raft_server()] 2021-04-16T23:04:57.838_589+00:00 [2c40] [INFO] peer 0: DC ID 0, 192.168.1.17:5047, voting member, 1 peer 0: DC ID 0, 192.168.1.17:5047, voting member, 1 peer 1: DC ID 0, 192.168.1.14:5047, voting member, 1 peer 2: DC ID 0, 192.168.1.13:5047, voting member, 1 my id: 0, voting_member num peers: 2 [raft_server.cxx:242, raft_server()] 2021-04-16T23:04:57.838_595+00:00 [2c40] [INFO] global manager does not exist. will use local thread for commit and append [raft_server.cxx:253, raft_server()] 2021-04-16T23:04:57.839_914+00:00 [2c40] [INFO] wait for HB, for 50 + [200, 400] ms [raft_server.cxx:280, raft_server()] 2021-04-16T23:04:57.840_004+00:00 [54a8] [INFO] bg append_entries thread initiated [handle_append_entries.cxx:47, append_entries_in_bg()] 2021-04-16T23:04:58.090_161+00:00 [58b5] [WARN] Election timeout, initiate leader election [handle_timeout.cxx:256, handle_election_timeout()] 2021-04-16T23:04:58.090_181+00:00 [58b5] [INFO] [PRIORITY] decay, target 1 -> 1, mine 1 [handle_priority.cxx:212, decay_target_priority()] 2021-04-16T23:04:58.090_200+00:00 [58b5] [INFO] [ELECTION TIMEOUT] current role: follower, log last term 0, state term 0, target p 1, my p 1, hb dead, pre-vote NOT done [handle_timeout.cxx:274, handle_election_timeout()] 2021-04-16T23:04:58.090_207+00:00 [58b5] [INFO] [PRE-VOTE INIT] my id 0, my role follower, term 0, log idx 0, log term 0, priority (target 1 / mine 1) [handle_vote.cxx:140, request_prevote()] 2021-04-16T23:04:58.092_035+00:00 [58b5] [WARN] peer (1) response error: failed to connect to peer 1, 192.168.1.14:5047, error 111 [raft_server.cxx:719, handle_peer_resp()] 2021-04-16T23:04:58.092_072+00:00 [58b5] [WARN] peer (2) response error: failed to connect to peer 2, 192.168.1.13:5047, error 111 [raft_server.cxx:719, handle_peer_resp()] 2021-04-16T23:04:58.317_192+00:00 [58b5] [WARN] Election timeout, initiate leader election [handle_timeout.cxx:256, handle_election_timeout()] 2021-04-16T23:04:58.317_212+00:00 [58b5] [INFO] [PRIORITY] decay, target 1 -> 1, mine 1 [handle_priority.cxx:212, decay_target_priority()] 2021-04-16T23:04:58.317_245+00:00 [58b5] [INFO] [ELECTION TIMEOUT] current role: follower, log last term 0, state term 0, target p 1, my p 1, hb dead, pre-vote NOT done [handle_timeout.cxx:274, handle_election_timeout()] 2021-04-16T23:04:58.317_252+00:00 [58b5] [INFO] reset RPC client for peer 2 [handle_vote.cxx:79, request_prevote()] 2021-04-16T23:04:58.317_280+00:00 [58b5] [INFO] reset RPC client for peer 1 [handle_vote.cxx:79, request_prevote()] 2021-04-16T23:04:58.317_285+00:00 [58b5] [WARN] total 1 nodes (including this node) responded for pre-vote (term 0, live 0, dead 1), at least 2 nodes should respond. failure count 1 [handle_vote.cxx:100, request_prevote()] 2021-04-16T23:04:58.317_289+00:00 [58b5] [INFO] [PRE-VOTE INIT] my id 0, my role follower, term 0, log idx 0, log term 0, priority (target 1 / mine 1) [handle_vote.cxx:140, request_prevote()] 2021-04-16T23:04:58.318_362+00:00 [58b5] [WARN] peer (1) response error: failed to connect to peer 1, 192.168.1.14:5047, error 111 [raft_server.cxx:719, handle_peer_resp()] 2021-04-16T23:04:58.318_431+00:00 [58b5] [WARN] peer (2) response error: failed to connect to peer 2, 192.168.1.13:5047, error 111 [raft_server.cxx:719, handle_peer_resp()] 2021-04-16T23:04:58.668_398+00:00 [58b5] [WARN] Election timeout, initiate leader election [handle_timeout.cxx:256, handle_election_timeout()] 2021-04-16T23:04:58.668_416+00:00 [58b5] [INFO] [PRIORITY] decay, target 1 -> 1, mine 1 [handle_priority.cxx:212, decay_target_priority()] 2021-04-16T23:04:58.668_433+00:00 [58b5] [INFO] [ELECTION TIMEOUT] current role: follower, log last term 0, state term 0, target p 1, my p 1, hb dead, pre-vote NOT done [handle_timeout.cxx:274, handle_election_timeout()] 2021-04-16T23:04:58.668_440+00:00 [58b5] [INFO] reset RPC client for peer 2 [handle_vote.cxx:79, request_prevote()] 2021-04-16T23:04:58.668_473+00:00 [58b5] [INFO] reset RPC client for peer 1 [handle_vote.cxx:79, request_prevote()] 2021-04-16T23:04:58.668_478+00:00 [58b5] [WARN] total 1 nodes (including this node) responded for pre-vote (term 0, live 0, dead 1), at least 2 nodes should respond. failure count 2 [handle_vote.cxx:100, request_prevote()] 2021-04-16T23:04:58.668_482+00:00 [58b5] [INFO] [PRE-VOTE INIT] my id 0, my role follower, term 0, log idx 0, log term 0, priority (target 1 / mine 1) [handle_vote.cxx:140, request_prevote()] 2021-04-16T23:04:58.669_467+00:00 [58b5] [WARN] peer (1) response error: failed to connect to peer 1, 192.168.1.14:5047, error 111 [raft_server.cxx:719, handle_peer_resp()] 2021-04-16T23:04:58.669_503+00:00 [58b5] [WARN] peer (2) response error: failed to connect to peer 2, 192.168.1.13:5047, error 111 [raft_server.cxx:719, handle_peer_resp()] 2021-04-16T23:04:58.960_594+00:00 [58b5] [WARN] Election timeout, initiate leader election [handle_timeout.cxx:256, handle_election_timeout()] 2021-04-16T23:04:58.960_611+00:00 [58b5] [INFO] [PRIORITY] decay, target 1 -> 1, mine 1 [handle_priority.cxx:212, decay_target_priority()] 2021-04-16T23:04:58.960_627+00:00 [58b5] [INFO] [ELECTION TIMEOUT] current role: follower, log last term 0, state term 0, target p 1, my p 1, hb dead, pre-vote NOT done [handle_timeout.cxx:274, handle_election_timeout()] 2021-04-16T23:04:58.960_634+00:00 [58b5] [INFO] reset RPC client for peer 2 [handle_vote.cxx:79, request_prevote()] 2021-04-16T23:04:58.960_646+00:00 [58b5] [INFO] reset RPC client for peer 1 [handle_vote.cxx:79, request_prevote()] 2021-04-16T23:04:58.960_650+00:00 [58b5] [WARN] total 1 nodes (including this node) responded for pre-vote (term 0, live 0, dead 1), at least 2 nodes should respond. failure count 3 [handle_vote.cxx:100, request_prevote()] 2021-04-16T23:04:58.960_654+00:00 [58b5] [INFO] [PRE-VOTE INIT] my id 0, my role follower, term 0, log idx 0, log term 0, priority (target 1 / mine 1) [handle_vote.cxx:140, request_prevote()] 2021-04-16T23:04:58.961_694+00:00 [58b5] [WARN] peer (2) response error: failed to connect to peer 2, 192.168.1.13:5047, error 111 [raft_server.cxx:719, handle_peer_resp()] 2021-04-16T23:04:58.961_727+00:00 [58b5] [WARN] peer (1) response error: failed to connect to peer 1, 192.168.1.14:5047, error 111 [raft_server.cxx:719, handle_peer_resp()] 2021-04-16T23:04:59.267_718+00:00 [58b5] [WARN] Election timeout, initiate leader election [handle_timeout.cxx:256, handle_election_timeout()] 2021-04-16T23:04:59.267_755+00:00 [58b5] [INFO] [PRIORITY] decay, target 1 -> 1, mine 1 [handle_priority.cxx:212, decay_target_priority()] 2021-04-16T23:04:59.267_792+00:00 [58b5] [INFO] [ELECTION TIMEOUT] current role: follower, log last term 0, state term 0, target p 1, my p 1, hb dead, pre-vote NOT done [handle_timeout.cxx:274, handle_election_timeout()] 2021-04-16T23:04:59.267_798+00:00 [58b5] [INFO] reset RPC client for peer 2 [handle_vote.cxx:79, request_prevote()] 2021-04-16T23:04:59.267_836+00:00 [58b5] [INFO] reset RPC client for peer 1 [handle_vote.cxx:79, request_prevote()] 2021-04-16T23:04:59.267_840+00:00 [58b5] [WARN] total 1 nodes (including this node) responded for pre-vote (term 0, live 0, dead 1), at least 2 nodes should respond. failure count 4 [handle_vote.cxx:100, request_prevote()] 2021-04-16T23:04:59.267_844+00:00 [58b5] [INFO] [PRE-VOTE INIT] my id 0, my role follower, term 0, log idx 0, log term 0, priority (target 1 / mine 1) [handle_vote.cxx:140, request_prevote()] 2021-04-16T23:04:59.268_809+00:00 [58b5] [WARN] peer (1) response error: failed to connect to peer 1, 192.168.1.14:5047, error 111 [raft_server.cxx:719, handle_peer_resp()] 2021-04-16T23:04:59.268_843+00:00 [58b5] [WARN] peer (2) response error: failed to connect to peer 2, 192.168.1.13:5047, error 111 [raft_server.cxx:719, handle_peer_resp()] 2021-04-16T23:04:59.511_961+00:00 [58b5] [WARN] Election timeout, initiate leader election [handle_timeout.cxx:256, handle_election_timeout()] 2021-04-16T23:04:59.511_981+00:00 [58b5] [INFO] [PRIORITY] decay, target 1 -> 1, mine 1 [handle_priority.cxx:212, decay_target_priority()] 2021-04-16T23:04:59.511_997+00:00 [58b5] [INFO] [ELECTION TIMEOUT] current role: follower, log last term 0, state term 0, target p 1, my p 1, hb dead, pre-vote NOT done [handle_timeout.cxx:274, handle_election_timeout()] 2021-04-16T23:04:59.512_004+00:00 [58b5] [INFO] reset RPC client for peer 2 [handle_vote.cxx:79, request_prevote()] 2021-04-16T23:04:59.512_016+00:00 [58b5] [INFO] reset RPC client for peer 1 [handle_vote.cxx:79, request_prevote()] 2021-04-16T23:04:59.512_021+00:00 [58b5] [WARN] total 1 nodes (including this node) responded for pre-vote (term 0, live 0, dead 1), at least 2 nodes should respond. failure count 5 [handle_vote.cxx:100, request_prevote()] 2021-04-16T23:04:59.512_024+00:00 [58b5] [INFO] [PRE-VOTE INIT] my id 0, my role follower, term 0, log idx 0, log term 0, priority (target 1 / mine 1) [handle_vote.cxx:140, request_prevote()] 2021-04-16T23:04:59.513_016+00:00 [58b5] [WARN] peer (2) response error: failed to connect to peer 2, 192.168.1.13:5047, error 111 [raft_server.cxx:719, handle_peer_resp()] 2021-04-16T23:04:59.516_176+00:00 [58b5] [WARN] peer (1) response error: failed to connect to peer 1, 192.168.1.14:5047, error 111 [raft_server.cxx:719, handle_peer_resp()] 2021-04-16T23:04:59.721_202+00:00 [58b5] [WARN] Election timeout, initiate leader election [handle_timeout.cxx:256, handle_election_timeout()] 2021-04-16T23:04:59.721_244+00:00 [58b5] [INFO] [PRIORITY] decay, target 1 -> 1, mine 1 [handle_priority.cxx:212, decay_target_priority()] 2021-04-16T23:04:59.721_260+00:00 [58b5] [INFO] [ELECTION TIMEOUT] current role: follower, log last term 0, state term 0, target p 1, my p 1, hb dead, pre-vote NOT done [handle_timeout.cxx:274, handle_election_timeout()] 2021-04-16T23:04:59.721_283+00:00 [58b5] [INFO] reset RPC client for peer 2 [handle_vote.cxx:79, request_prevote()] 2021-04-16T23:04:59.721_295+00:00 [58b5] [INFO] reset RPC client for peer 1 [handle_vote.cxx:79, request_prevote()] 2021-04-16T23:04:59.721_300+00:00 [58b5] [WARN] total 1 nodes (including this node) responded for pre-vote (term 0, live 0, dead 1), at least 2 nodes should respond. failure count 6 [handle_vote.cxx:100, request_prevote()] 2021-04-16T23:04:59.721_320+00:00 [58b5] [INFO] [PRE-VOTE INIT] my id 0, my role follower, term 0, log idx 0, log term 0, priority (target 1 / mine 1) [handle_vote.cxx:140, request_prevote()] 2021-04-16T23:04:59.722_695+00:00 [58b5] [WARN] peer (1) response error: failed to connect to peer 1, 192.168.1.14:5047, error 111 [raft_server.cxx:719, handle_peer_resp()] 2021-04-16T23:04:59.722_728+00:00 [58b5] [WARN] peer (2) response error: failed to connect to peer 2, 192.168.1.13:5047, error 111 [raft_server.cxx:719, handle_peer_resp()] 2021-04-16T23:05:00.057_445+00:00 [58b5] [WARN] Election timeout, initiate leader election [handle_timeout.cxx:256, handle_election_timeout()] 2021-04-16T23:05:00.057_466+00:00 [58b5] [INFO] [PRIORITY] decay, target 1 -> 1, mine 1 [handle_priority.cxx:212, decay_target_priority()] 2021-04-16T23:05:00.057_482+00:00 [58b5] [INFO] [ELECTION TIMEOUT] current role: follower, log last term 0, state term 0, target p 1, my p 1, hb dead, pre-vote NOT done [handle_timeout.cxx:274, handle_election_timeout()] 2021-04-16T23:05:00.057_490+00:00 [58b5] [INFO] reset RPC client for peer 2 [handle_vote.cxx:79, request_prevote()] 2021-04-16T23:05:00.057_502+00:00 [58b5] [INFO] reset RPC client for peer 1 [handle_vote.cxx:79, request_prevote()] 2021-04-16T23:05:00.057_507+00:00 [58b5] [WARN] total 1 nodes (including this node) responded for pre-vote (term 0, live 0, dead 1), at least 2 nodes should respond. failure count 7 [handle_vote.cxx:100, request_prevote()] 2021-04-16T23:05:00.057_511+00:00 [58b5] [INFO] [PRE-VOTE INIT] my id 0, my role follower, term 0, log idx 0, log term 0, priority (target 1 / mine 1) [handle_vote.cxx:140, request_prevote()] 2021-04-16T23:05:00.058_656+00:00 [58b5] [WARN] peer (1) response error: failed to connect to peer 1, 192.168.1.14:5047, error 111 [raft_server.cxx:719, handle_peer_resp()] 2021-04-16T23:05:00.058_711+00:00 [58b5] [WARN] peer (2) response error: failed to connect to peer 2, 192.168.1.13:5047, error 111 [raft_server.cxx:719, handle_peer_resp()] 2021-04-16T23:05:00.393_608+00:00 [58b5] [WARN] Election timeout, initiate leader election [handle_timeout.cxx:256, handle_election_timeout()] 2021-04-16T23:05:00.393_641+00:00 [58b5] [INFO] [PRIORITY] decay, target 1 -> 1, mine 1 [handle_priority.cxx:212, decay_target_priority()] 2021-04-16T23:05:00.393_656+00:00 [58b5] [INFO] [ELECTION TIMEOUT] current role: follower, log last term 0, state term 0, target p 1, my p 1, hb dead, pre-vote NOT done [handle_timeout.cxx:274, handle_election_timeout()] 2021-04-16T23:05:00.393_662+00:00 [58b5] [INFO] reset RPC client for peer 2 [handle_vote.cxx:79, request_prevote()] 2021-04-16T23:05:00.393_673+00:00 [58b5] [INFO] reset RPC client for peer 1 [handle_vote.cxx:79, request_prevote()] 2021-04-16T23:05:00.393_677+00:00 [58b5] [WARN] total 1 nodes (including this node) responded for pre-vote (term 0, live 0, dead 1), at least 2 nodes should respond. failure count 8 [handle_vote.cxx:100, request_prevote()] 2021-04-16T23:05:00.393_680+00:00 [58b5] [INFO] [PRE-VOTE INIT] my id 0, my role follower, term 0, log idx 0, log term 0, priority (target 1 / mine 1) [handle_vote.cxx:140, request_prevote()] 2021-04-16T23:05:00.394_671+00:00 [58b5] [WARN] peer (1) response error: failed to connect to peer 1, 192.168.1.14:5047, error 111 [raft_server.cxx:719, handle_peer_resp()] 2021-04-16T23:05:00.394_720+00:00 [58b5] [WARN] peer (2) response error: failed to connect to peer 2, 192.168.1.13:5047, error 111 [raft_server.cxx:719, handle_peer_resp()] 2021-04-16T23:05:00.780_792+00:00 [58b5] [WARN] Election timeout, initiate leader election [handle_timeout.cxx:256, handle_election_timeout()] 2021-04-16T23:05:00.780_812+00:00 [58b5] [INFO] [PRIORITY] decay, target 1 -> 1, mine 1 [handle_priority.cxx:212, decay_target_priority()] 2021-04-16T23:05:00.780_828+00:00 [58b5] [INFO] [ELECTION TIMEOUT] current role: follower, log last term 0, state term 0, target p 1, my p 1, hb dead, pre-vote NOT done [handle_timeout.cxx:274, handle_election_timeout()] 2021-04-16T23:05:00.780_835+00:00 [58b5] [INFO] reset RPC client for peer 2 [handle_vote.cxx:79, request_prevote()] 2021-04-16T23:05:00.780_847+00:00 [58b5] [INFO] reset RPC client for peer 1 [handle_vote.cxx:79, request_prevote()] 2021-04-16T23:05:00.780_852+00:00 [58b5] [WARN] total 1 nodes (including this node) responded for pre-vote (term 0, live 0, dead 1), at least 2 nodes should respond. failure count 9 [handle_vote.cxx:100, request_prevote()] 2021-04-16T23:05:00.780_855+00:00 [58b5] [INFO] [PRE-VOTE INIT] my id 0, my role follower, term 0, log idx 0, log term 0, priority (target 1 / mine 1) [handle_vote.cxx:140, request_prevote()] 2021-04-16T23:05:00.782_034+00:00 [58b5] [WARN] peer (1) response error: failed to connect to peer 1, 192.168.1.14:5047, error 111 [raft_server.cxx:719, handle_peer_resp()] 2021-04-16T23:05:00.782_084+00:00 [58b5] [WARN] peer (2) response error: failed to connect to peer 2, 192.168.1.13:5047, error 111 [raft_server.cxx:719, handle_peer_resp()] 2021-04-16T23:05:01.058_016+00:00 [58b5] [WARN] Election timeout, initiate leader election [handle_timeout.cxx:256, handle_election_timeout()] 2021-04-16T23:05:01.058_037+00:00 [58b5] [INFO] [PRIORITY] decay, target 1 -> 1, mine 1 [handle_priority.cxx:212, decay_target_priority()] 2021-04-16T23:05:01.058_055+00:00 [58b5] [INFO] [ELECTION TIMEOUT] current role: follower, log last term 0, state term 0, target p 1, my p 1, hb dead, pre-vote NOT done [handle_timeout.cxx:274, handle_election_timeout()] 2021-04-16T23:05:01.058_062+00:00 [58b5] [INFO] reset RPC client for peer 2 [handle_vote.cxx:79, request_prevote()] 2021-04-16T23:05:01.058_075+00:00 [58b5] [INFO] reset RPC client for peer 1 [handle_vote.cxx:79, request_prevote()] 2021-04-16T23:05:01.058_080+00:00 [58b5] [WARN] total 1 nodes (including this node) responded for pre-vote (term 0, live 0, dead 1), at least 2 nodes should respond. failure count 10 [handle_vote.cxx:100, request_prevote()] 2021-04-16T23:05:01.058_085+00:00 [58b5] [INFO] [PRE-VOTE INIT] my id 0, my role follower, term 0, log idx 0, log term 0, priority (target 1 / mine 1) [handle_vote.cxx:140, request_prevote()] 2021-04-16T23:05:01.059_213+00:00 [58b5] [WARN] peer (1) response error: failed to connect to peer 1, 192.168.1.14:5047, error 111 [raft_server.cxx:719, handle_peer_resp()] 2021-04-16T23:05:01.059_246+00:00 [58b5] [WARN] peer (2) response error: failed to connect to peer 2, 192.168.1.13:5047, error 111 [raft_server.cxx:719, handle_peer_resp()] 2021-04-16T23:05:01.362_229+00:00 [58b5] [WARN] Election timeout, initiate leader election [handle_timeout.cxx:256, handle_election_timeout()] 2021-04-16T23:05:01.362_245+00:00 [58b5] [INFO] [PRIORITY] decay, target 1 -> 1, mine 1 [handle_priority.cxx:212, decay_target_priority()] 2021-04-16T23:05:01.362_277+00:00 [58b5] [INFO] [ELECTION TIMEOUT] current role: follower, log last term 0, state term 0, target p 1, my p 1, hb dead, pre-vote NOT done [handle_timeout.cxx:274, handle_election_timeout()] 2021-04-16T23:05:01.362_283+00:00 [58b5] [INFO] reset RPC client for peer 2 [handle_vote.cxx:79, request_prevote()] 2021-04-16T23:05:01.362_294+00:00 [58b5] [INFO] reset RPC client for peer 1 [handle_vote.cxx:79, request_prevote()] 2021-04-16T23:05:01.362_299+00:00 [58b5] [WARN] total 1 nodes (including this node) responded for pre-vote (term 0, live 0, dead 1), at least 2 nodes should respond. failure count 11 [handle_vote.cxx:100, request_prevote()] 2021-04-16T23:05:01.362_302+00:00 [58b5] [INFO] [PRE-VOTE INIT] my id 0, my role follower, term 0, log idx 0, log term 0, priority (target 1 / mine 1) [handle_vote.cxx:140, request_prevote()] 2021-04-16T23:05:01.363_345+00:00 [58b5] [WARN] peer (1) response error: failed to connect to peer 1, 192.168.1.14:5047, error 111 [raft_server.cxx:719, handle_peer_resp()] 2021-04-16T23:05:01.363_662+00:00 [58b5] [WARN] peer (2) response error: failed to connect to peer 2, 192.168.1.13:5047, error 111 [raft_server.cxx:719, handle_peer_resp()] 2021-04-16T23:05:01.729_416+00:00 [58b5] [WARN] Election timeout, initiate leader election [handle_timeout.cxx:256, handle_election_timeout()] 2021-04-16T23:05:01.729_437+00:00 [58b5] [INFO] [PRIORITY] decay, target 1 -> 1, mine 1 [handle_priority.cxx:212, decay_target_priority()] 2021-04-16T23:05:01.729_454+00:00 [58b5] [INFO] [ELECTION TIMEOUT] current role: follower, log last term 0, state term 0, target p 1, my p 1, hb dead, pre-vote NOT done [handle_timeout.cxx:274, handle_election_timeout()] 2021-04-16T23:05:01.729_461+00:00 [58b5] [INFO] reset RPC client for peer 2 [handle_vote.cxx:79, request_prevote()] 2021-04-16T23:05:01.729_474+00:00 [58b5] [INFO] reset RPC client for peer 1 [handle_vote.cxx:79, request_prevote()] 2021-04-16T23:05:01.729_479+00:00 [58b5] [WARN] total 1 nodes (including this node) responded for pre-vote (term 0, live 0, dead 1), at least 2 nodes should respond. failure count 12 [handle_vote.cxx:100, request_prevote()] 2021-04-16T23:05:01.729_483+00:00 [58b5] [INFO] [PRE-VOTE INIT] my id 0, my role follower, term 0, log idx 0, log term 0, priority (target 1 / mine 1) [handle_vote.cxx:140, request_prevote()] 2021-04-16T23:05:01.730_463+00:00 [58b5] [WARN] peer (1) response error: failed to connect to peer 1, 192.168.1.14:5047, error 111 [raft_server.cxx:719, handle_peer_resp()] 2021-04-16T23:05:01.730_624+00:00 [58b5] [WARN] peer (2) response error: failed to connect to peer 2, 192.168.1.13:5047, error 111 [raft_server.cxx:719, handle_peer_resp()] 2021-04-16T23:05:01.935_628+00:00 [58b5] [WARN] Election timeout, initiate leader election [handle_timeout.cxx:256, handle_election_timeout()] 2021-04-16T23:05:01.935_642+00:00 [58b5] [INFO] [PRIORITY] decay, target 1 -> 1, mine 1 [handle_priority.cxx:212, decay_target_priority()] 2021-04-16T23:05:01.935_674+00:00 [58b5] [INFO] [ELECTION TIMEOUT] current role: follower, log last term 0, state term 0, target p 1, my p 1, hb dead, pre-vote NOT done [handle_timeout.cxx:274, handle_election_timeout()] 2021-04-16T23:05:01.935_681+00:00 [58b5] [INFO] reset RPC client for peer 2 [handle_vote.cxx:79, request_prevote()] 2021-04-16T23:05:01.935_693+00:00 [58b5] [INFO] reset RPC client for peer 1 [handle_vote.cxx:79, request_prevote()] 2021-04-16T23:05:01.935_698+00:00 [58b5] [WARN] total 1 nodes (including this node) responded for pre-vote (term 0, live 0, dead 1), at least 2 nodes should respond. failure count 13 [handle_vote.cxx:100, request_prevote()] 2021-04-16T23:05:01.935_701+00:00 [58b5] [INFO] [PRE-VOTE INIT] my id 0, my role follower, term 0, log idx 0, log term 0, priority (target 1 / mine 1) [handle_vote.cxx:140, request_prevote()] 2021-04-16T23:05:01.936_682+00:00 [58b5] [WARN] peer (1) response error: failed to connect to peer 1, 192.168.1.14:5047, error 111 [raft_server.cxx:719, handle_peer_resp()] 2021-04-16T23:05:01.936_751+00:00 [58b5] [WARN] peer (2) response error: failed to connect to peer 2, 192.168.1.13:5047, error 111 [raft_server.cxx:719, handle_peer_resp()] 2021-04-16T23:05:02.145_845+00:00 [58b5] [WARN] Election timeout, initiate leader election [handle_timeout.cxx:256, handle_election_timeout()] 2021-04-16T23:05:02.145_871+00:00 [58b5] [INFO] [PRIORITY] decay, target 1 -> 1, mine 1 [handle_priority.cxx:212, decay_target_priority()] 2021-04-16T23:05:02.145_893+00:00 [58b5] [INFO] [ELECTION TIMEOUT] current role: follower, log last term 0, state term 0, target p 1, my p 1, hb dead, pre-vote NOT done [handle_timeout.cxx:274, handle_election_timeout()] 2021-04-16T23:05:02.145_900+00:00 [58b5] [INFO] reset RPC client for peer 2 [handle_vote.cxx:79, request_prevote()] 2021-04-16T23:05:02.145_912+00:00 [58b5] [INFO] reset RPC client for peer 1 [handle_vote.cxx:79, request_prevote()] 2021-04-16T23:05:02.145_917+00:00 [58b5] [WARN] total 1 nodes (including this node) responded for pre-vote (term 0, live 0, dead 1), at least 2 nodes should respond. failure count 14 [handle_vote.cxx:100, request_prevote()] 2021-04-16T23:05:02.145_921+00:00 [58b5] [INFO] [PRE-VOTE INIT] my id 0, my role follower, term 0, log idx 0, log term 0, priority (target 1 / mine 1) [handle_vote.cxx:140, request_prevote()] 2021-04-16T23:05:02.146_956+00:00 [58b5] [WARN] peer (1) response error: failed to connect to peer 1, 192.168.1.14:5047, error 111 [raft_server.cxx:719, handle_peer_resp()] 2021-04-16T23:05:02.146_987+00:00 [58b5] [WARN] peer (2) response error: failed to connect to peer 2, 192.168.1.13:5047, error 111 [raft_server.cxx:719, handle_peer_resp()] 2021-04-16T23:05:02.452_005+00:00 [58b5] [WARN] Election timeout, initiate leader election [handle_timeout.cxx:256, handle_election_timeout()] 2021-04-16T23:05:02.452_026+00:00 [58b5] [INFO] [PRIORITY] decay, target 1 -> 1, mine 1 [handle_priority.cxx:212, decay_target_priority()] 2021-04-16T23:05:02.452_044+00:00 [58b5] [INFO] [ELECTION TIMEOUT] current role: follower, log last term 0, state term 0, target p 1, my p 1, hb dead, pre-vote NOT done [handle_timeout.cxx:274, handle_election_timeout()] 2021-04-16T23:05:02.452_051+00:00 [58b5] [INFO] reset RPC client for peer 2 [handle_vote.cxx:79, request_prevote()] 2021-04-16T23:05:02.452_063+00:00 [58b5] [INFO] reset RPC client for peer 1 [handle_vote.cxx:79, request_prevote()] 2021-04-16T23:05:02.452_069+00:00 [58b5] [WARN] total 1 nodes (including this node) responded for pre-vote (term 0, live 0, dead 1), at least 2 nodes should respond. failure count 15 [handle_vote.cxx:100, request_prevote()] 2021-04-16T23:05:02.452_073+00:00 [58b5] [INFO] [PRE-VOTE INIT] my id 0, my role follower, term 0, log idx 0, log term 0, priority (target 1 / mine 1) [handle_vote.cxx:140, request_prevote()] 2021-04-16T23:05:02.452_987+00:00 [58b5] [WARN] peer (1) response error: failed to connect to peer 1, 192.168.1.14:5047, error 111 [raft_server.cxx:719, handle_peer_resp()] 2021-04-16T23:05:02.453_033+00:00 [58b5] [WARN] peer (2) response error: failed to connect to peer 2, 192.168.1.13:5047, error 111 [raft_server.cxx:719, handle_peer_resp()] 2021-04-16T23:05:02.786_173+00:00 [58b5] [WARN] Election timeout, initiate leader election [handle_timeout.cxx:256, handle_election_timeout()] 2021-04-16T23:05:02.786_194+00:00 [58b5] [INFO] [PRIORITY] decay, target 1 -> 1, mine 1 [handle_priority.cxx:212, decay_target_priority()] 2021-04-16T23:05:02.786_210+00:00 [58b5] [INFO] [ELECTION TIMEOUT] current role: follower, log last term 0, state term 0, target p 1, my p 1, hb dead, pre-vote NOT done [handle_timeout.cxx:274, handle_election_timeout()] 2021-04-16T23:05:02.786_218+00:00 [58b5] [INFO] reset RPC client for peer 2 [handle_vote.cxx:79, request_prevote()] 2021-04-16T23:05:02.786_231+00:00 [58b5] [INFO] reset RPC client for peer 1 [handle_vote.cxx:79, request_prevote()] 2021-04-16T23:05:02.786_237+00:00 [58b5] [WARN] total 1 nodes (including this node) responded for pre-vote (term 0, live 0, dead 1), at least 2 nodes should respond. failure count 16 [handle_vote.cxx:100, request_prevote()] 2021-04-16T23:05:02.786_241+00:00 [58b5] [INFO] [PRE-VOTE INIT] my id 0, my role follower, term 0, log idx 0, log term 0, priority (target 1 / mine 1) [handle_vote.cxx:140, request_prevote()] 2021-04-16T23:05:02.787_729+00:00 [58b5] [WARN] peer (2) response error: failed to connect to peer 2, 192.168.1.13:5047, error 111 [raft_server.cxx:719, handle_peer_resp()] 2021-04-16T23:05:02.787_763+00:00 [58b5] [WARN] peer (1) response error: failed to connect to peer 1, 192.168.1.14:5047, error 111 [raft_server.cxx:719, handle_peer_resp()]

[greensky00]

A Raft server is ready (initialized) and functional when it becomes either leader or follower. Since you put 3 servers to the pre-defined cluster config, a single server can be neither leader (it cannot reach a consensus to be elected, at least 2 votes are required) nor follower (no leader exists).

You can just comment out below the initialization checking logic in that example.

NuRaft/examples/example_common.hxx

Lines 203 to 214 in 8699f3b

	// Wait until Raft server is ready (upto 5 seconds).
	const size_t MAX_TRY = 20;
	std::cout << "init Raft instance ";
	for (size_t ii=0; ii<MAX_TRY; ++ii) {
	if (stuff.raft_instance_->is_initialized()) {
	std::cout << " done" << std::endl;
	return;
	}
	std::cout << ".";
	fflush(stdout);
	TestSuite::sleep_ms(250);
	}

Each server keeps attempting leader election or waiting for the heartbeat from a leader, hence as soon as two servers start running, they form a working Raft group and will be ready.

[kishorekrd]

That worked. Thanks. I see that list command lists the configured servers in the cluster, but they are not necessarily up at that time.
How to list the servers that are only up?
When do leader declares a follower as down?
Is there any API to get this details?
How can I add a call-back function to be called when any follower goes down or comes up?