jdraftkeeper / raftkeeper Goto Github PK

RaftKeeper is a high-performance distributed consensus service.

License: Apache License 2.0

CMake 0.45% C++ 85.40% C 0.85% Assembly 12.46% Shell 0.05% Java 0.09% Dockerfile 0.02% Python 0.68% Perl 0.01%

raft kv-storage coordination cpp metadata high-performance

raftkeeper's Introduction

RaftKeeper is a high-performance distributed consensus service. It is fully compatible with Zookeeper and can be accessed through the Zookeeper client. It implements most of the functions of Zookeeper (except: Container node, TTL node, quota etc.) and provides some additional functions, such as more monitoring indicators, manual Leader switching and so on.

RaftKeeper provides same consistency guarantee:

Responses must be returned in order in one session.
All committed write requests must be handled in order, across all sessions.

RaftKeeper data resides in memory and provides snapshot + operation log data persistence capabilities. The execution framework adopts pipeline and batch execution methods to greatly improve system throughput.

The main features of RaftKeeper are its performance and query stability.

The write-request throughput is twice as high as Zookeeper
TP99 is smoother than Zookeeper

See benchmark for details.

RaftKeeper is derived from ClickHouse and take NuRaft as Raft implementation. We really appreciate the excellent work of the ClickHouse and NuRaft teams.

How to start?

1. Build RaftKeeper

Build on Ubuntu

Requirement: Ubuntu 20.04+, Clang 13+(17 is recommended), Cmake 3.20+

# install tools
sudo apt-get install cmake llvm-17
 
# clone project
git clone https://github.com/JDRaftKeeper/RaftKeeper.git
git submodule sync && git submodule update --init --recursive
 
# build project
export CC=`which clang-17` CXX=`which clang++-17`
cd RaftKeeper && /bin/bash build.sh

# build for ClickHouse usage (for ClickHouse client is slightly incompatible with zookeeper)
cd RaftKeeper && /bin/bash build.sh 'clickhouse'

# now you can find built files in director 'build/'

Now RaftKeeper support build on Linux and Mac OX, details see how-to-build

2. Deploy RaftKeeper

To deploy a RaftKeeper cluster you can see how-to-deploy.

3. Access RaftKeeper

You can use ZooKeeper's shell client zkCli.sh to access to RaftKeeper, or you can use Java, python or C ZooKeeper clients to access it. The following is a zkCli.sh demo

./zkCli.sh -server localhost:8101

Also, RaftKeeper supports Zookeeper's 4lw command. 4lw command provide the ablility to monitor and manage RaftKeeper, details see how-to-monitor-and-manage

How to migrate from Zookeeper?

RaftKeeper provides a tool to translate Zookeeper data to RaftKeeper format. So you can simply move the translated data into RaftKeeper, detail is in how-to-migrate-from-zookeeper.

raftkeeper's People

Contributors

Stargazers

Watchers

raftkeeper's Issues

Add 4lw command uptime

Description

Support for obtaining the start time, it is a useful feature.

How to achieve

by 4lw command

Under ubuntu22.04 version, compilation and packaging failed

Build on Ubuntu failed，Here are my steps，What should I do to build successfully

environmental information

System version: Ubuntu 22.04 LTS
clang version: Ubuntu clang version 14.0.0-1ubuntu1
cmake version: cmake version 3.22.1

step1

# install tools
sudo apt-get install cmake llvm-13

step2

# clone project
git clone https://github.com/JDRaftKeeper/RaftKeeper.git

step3

cd RaftKeeper && git submodule sync && git submodule update --init --recursive

step4

# build project
export CC=`which clang` CXX=`which clang++`

step5

/bin/bash build.sh

Step 5 executes an error report, modifies the CMakeLists.txt in the root directory of RaftKeeper, and adds the following configuration at the file entry

set(CMAKE_THREAD_LIBS_INIT "-lpthread")
set(CMAKE_HAVE_THREADS_LIBRARY 1)
set(CMAKE_USE_WIN32_THREADS_INIT 0)
set(CMAKE_USE_PTHREADS_INIT 1)
set(THREADS_PREFER_PTHREAD_FLAG ON)

Re-execute after modification

/bin/bash build.sh

Request forwarder introduce request timeout mechanism

Description

Request forwarder introduce request timeout mechanism.

If the data has been successfully sent to the Linux kernel buffer during forwarding, but the underlying network is unstable, the application layer will not be aware of it. A timeout mechanism needs to be introduced.

But when the network times out, the peer may have successfully received and processed the request. We cannot guarantee this situation.

/cc @JackyWoo

Code of Conduct
The Code of Conduct helps create a safe space for everyone. We require that everyone agrees to it.

[ ] I agree to follow this project's Code of Conduct *

Use List for client Request Storage

Description

I notice that in RequestProcessor.h, a vector RequestForSessions is used to store all the non-write request for a sepecific session. There are three operations used most in this vector : push_back, erase, assignment(=). New request is pushed back, while the finished request is assigned and erased. In my opinion, The usage of RequestForSessions is not suitable in these senarios. erase operation in vector is a relatively heavy operation, with a time complexity of O(n). In scenarios where there are more reads than writes, the performance issues of this operation will be further magnified.

I suggest replacing the vector with some other datastructure to improve performance in high-concurrency scenarios.

Are you willing to submit PR?

Yes.

Dead lock when set log level to trace.

RaftKeeper version

v2.0.1

Description

Poco::SplitterChannel::log takes a non-recursive mutex and invoke twice in an invokation chain.

Thread 11 (Thread 0x7f24e5fff700 (LWP 69)):
#0  0x00007f24e9f4d54d in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00007f24e9f48e9b in _L_lock_883 () from /lib64/libpthread.so.0
#2  0x00007f24e9f48d68 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x00000000007298c3 in Poco::ScopedLock<Poco::FastMutex>::ScopedLock(Poco::FastMutex&) ()
#4  0x0000000000a4569d in Poco::SplitterChannel::log(Poco::Message const&) ()
#5  0x00000000006ec7d6 in MemoryTracker::logMemoryUsage(long) const ()
#6  0x00000000006ecc04 in MemoryTracker::alloc(long) ()
#7  0x00000000006da62b in operator new(unsigned long) ()
#8  0x0000000000bfac17 in std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >::append(unsigned long, char) ()
#9  0x00000000006e0099 in RK::WriteBufferFromVector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >::nextImpl() ()
#10 0x00000000006e0326 in RK::WriteBuffer::write(char const*, unsigned long) ()
#11 0x00000000007410c1 in OwnPatternFormatter::formatExtended(RK::ExtendedLogMessage const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >&) ()
#12 0x000000000074036b in RK::OwnFormattingChannel::logExtended(RK::ExtendedLogMessage const&) ()
#13 0x0000000000740491 in RK::OwnFormattingChannel::log(Poco::Message const&) ()
#14 0x0000000000a456c1 in Poco::SplitterChannel::log(Poco::Message const&) ()
#15 0x00000000007e1ab6 in void std::__1::__function::__policy_invoker<void ()>::__call_impl<std::__1::__function::__default_alloc_func<RK::NuRaftStateMachine::NuRaftStateMachine(RK::ThreadSafeQueue<RK::KeeperStore::ResponseForSession, std::__1::deque<RK::KeeperStore::ResponseForSession, std::__1::allocator<RK::KeeperStore::ResponseForSession> > >&, std::__1::shared_ptr<RK::RaftSettings> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >&, unsigned int, unsigned int, std::__1::mutex&, std::__1::unordered_map<long, std::__1::shared_ptr<std::__1::condition_variable>, std::__1::hash<long>, std::__1::equal_to<long>, std::__1::allocator<std::__1::pair<long const, std::__1::shared_ptr<std::__1::condition_variable> > > >&, std::__1::shared_ptr<nuraft::log_store>, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, unsigned int, std::__1::shared_ptr<RK::RequestProcessor>)::$_0, void ()> >(std::__1::__function::__policy_storage const*) ()
#16 0x00000000007259b9 in ThreadPoolImpl<ThreadFromGlobalPool>::worker(std::__1::__list_iterator<ThreadFromGlobalPool, void*>) ()
#17 0x0000000000727418 in ThreadFromGlobalPool::ThreadFromGlobalPool<bool ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<bool>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::{lambda()#3}>(bool ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<bool>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::{lambda()#3}&&)::{lambda()#1}::operator()() ()
#18 0x00000000007242aa in ThreadPoolImpl<std::__1::thread>::worker(std::__1::__list_iterator<std::__1::thread, void*>) ()
#19 0x000000000072606e in void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::{lambda()#3}> >(std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::{lambda()#3}>) ()
#20 0x00007f24e9f46ea5 in start_thread () from /lib64/libpthread.so.0
#21 0x00007f24ea763b0d in clone () from /lib64/libc.so.6

How to reproduce

Set up a RaftKeeper process
Create at lest 100m data and some of their path are longer than 100 bytes
Set RaftKeeper log level to trace
Restart RaftKeeper

Add sanitizer checking to CI

If this doesn’t look right, choose a different type.

Description

Run integration tests with sanitizer in CI. You can build binary by adding cmake option SANITIZE.

cmake -DSANITIZE=memory

Possible values:

RAFT is shutting down , when replay log request about Multi ops mixing of read and write operations

RaftKeeper version

v2.0

Description

Can’t start raft, after set multi ops mixing of read and write operations

2023.10.12 14:26:43.519224 [ 40740 ] <Trace> KeeperStateMachine: Replay log request, session 0xa, request Multi, Create, xid 0, path /test_multitransactions, data , is_ephemeral 0, is_sequential 0, Get, xid 0, path /test_multitransactions2, Get, xid 0, path /test_multitransactions333, Get, xid 0, path /test/test_multitransactions333
2023.10.12 14:26:43.519229 [ 40740 ] <Trace> KeeperStore: [process request]SessionID/xid #0xa#1, opnum Multi
2023.10.12 14:26:43.519402 [ 40740 ] <Information> RaftTaskManager: Shutting down task manager
2023.10.12 14:26:43.619059 [ 40743 ] <Information> FileLogStore: shutdown background raft log fsync thread.
2023.10.12 14:26:43.619172 [ 40740 ] <Error> Application: RK::Exception: Illegal mixing of read and write operations in multi request
2023.10.12 14:26:43.619186 [ 40740 ] <Information> Application: Shutting down RaftKeeper.
2023.10.12 14:26:43.619190 [ 40740 ] <Debug> Application: Uninitializing subsystem: Logging Subsystem
2023.10.12 14:26:43.619269 [ 40742 ] <Trace> BaseDaemon: Received signal -2
2023.10.12 14:26:43.619303 [ 40742 ] <Information> BaseDaemon: Stop SignalListener thread

How to reproduce

Send a request mix of read and write operations to raft
restart raft

Clang-format all source files

Description

There exists a .clang-format file, but not all source file is clang-formatted. We can do clang-format on every single RaftKeeper source file.

Are you willing to submit PR?

yes.

Request Processor endless loop.

RaftKeeper version

v2.0.1

Description

Has 18 error requests
2023.10.09 21:30:50.817469 [ 44 ] <Warning> RequestProcessor: Try find error request session 0xc0a, xid 3727471, error code -32768
2023.10.09 21:30:50.817500 [ 44 ] <Warning> RequestProcessor: Session 0xc0a, no pending requests
2023.10.09 21:30:50.817529 [ 44 ] <Warning> RequestProcessor: Not found error request session 0xc0a, xid 3727471 from pending queue. Maybe it is still in the request queue and will be processed next time
2023.10.09 21:30:50.817660 [ 44 ] <Warning> RequestProcessor: Has 18 error requests
2023.10.09 21:30:50.817690 [ 44 ] <Warning> RequestProcessor: Try find error request session 0xc0a, xid 3727471, error code -32768
2023.10.09 21:30:50.817722 [ 44 ] <Warning> RequestProcessor: Session 0xc0a, no pending requests
2023.10.09 21:30:50.817752 [ 44 ] <Warning> RequestProcessor: Not found error request session 0xc0a, xid 3727471 from pending queue. Maybe it is still in the request queue and will be processed next time
2023.10.09 21:30:50.818028 [ 44 ] <Warning> RequestProcessor: Has 18 error requests
...

This error will cause high CPU usage.

How to reproduce

Add names to threads.

Description

Add names to threads for debugging.

Are you willing to submit PR?

Install code style tools fail in CI

RaftKeeper version

Description

CI code style checking fails with erros:

E: Failed to fetch [http://azure.archive.ubuntu.com/ubuntu/pool/main/p/python3-stdlib-extensions/python3-tk_3.10.6-1%7e22.04_amd64.deb](http://azure.archive.ubuntu.com/ubuntu/pool/main/p/python3-stdlib-extensions/python3-tk_3.10.6-1~22.04_amd64.deb)  404  Not Found [IP: 40.119.46.219 80]
E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?

Support multi-read requests.

Describe your ideas

Support multi-read requests in which can not include write requests.

Solution

Poco library epoll_wait causes deadlock

Problems and issues with code of RaftKeeper. If this doesn’t look right, choose a choose a different type.

Fields marked with an asterisk (*) are required.

Thank you very much for submitting feedback to RaftKeeper to help develop better.

Search before asking *
Please make sure to search in the issues first to see whether the same issue was reported already.

[ ] I had searched in the issues and found no similar issues.

Version *

The lastest commit bf5f6c1

Describe the bug *
Cause the client and the server, and the server forwards the network to timeout.

Code of Conduct
The Code of Conduct helps create a safe space for everyone. We require that everyone agrees to it.

[ ] I agree to follow this project's Code of Conduct*

Tests `test_multinode_simple` is flaky

Problems and issues with code of RaftKeeper. If this doesn’t look right, choose a choose a different type.

RaftKeeper version

all

How to reproduce

Just trigger tests test_multinode_simple in CI

Analysis

tests test_multinode_simple in CI fails occasionally, but success in my local environment.

test_multinode_simple/test.py::test_session_expiration will partition node3 with node1 and node2 with iptables within which time client will try to connect to node1 and will cause creating new session which will lead node1 forwarding session creating request to partitioned leader.

Because of network partition package will be abandoned by iptables and NuRaft auto forwarding does not set connection timeout, the IO handing threads will be stuck and lead to node1 server hung up.

Why CI fails and local enviromnent will not?

Because my local machine has 96 CPU cores and CI only has 2, it means my local machine environment has more IO threads. When client try to connection to node1, it will stick one IO thread by every attempt. At the mean while the backgroud leader selection will complete at which moment the fowarding will not send to node3 and no suck will happen. So everything will be fine.

Why fails occasionally?

Because some time node3 is not leader and no forwarding to partitioned node happens.

How to fix？

Set connection timeout when follower forwarding request to leader (change happens in NuRaft)
New Forwarder (Working by @nicelulu ) : discard NuRaft forwarding.

[proposal] support apache Puslar

If this doesn’t look right, choose a different type.

Describe your ideas

Support work with apache Pulsar. see #34

Tasks:

Add AddWatch command
Add removeWatch command

Could not pass CI, stuck on build-and-test

Problems and issues with code of RaftKeeper. If this doesn’t look right, choose a choose a different type.

Describe your doubt

I pass all integration tests on my machine, but fails everytime on build-and-test.

Add more monitor metrics

Use case

RaftKeeper provides monitor by 4lw command mntr, but metrics are measured in whole process live time, which is not obvious enough.

Describe your idea

Add some metrics who can be measureed by 10s and 60s to mntr, such as:

zk_avg_latency_10s
zk_max_latency_10s
zk_min_latency_10s
zk_packets_received_10s
zk_packets_sent_10s

zk_avg_latency_60s
zk_max_latency_60s
zk_min_latency_60s
zk_packets_received_60s
zk_packets_sent_60s

Release v2.0.0 has SAXParseException

Problems and issues with code of RaftKeeper. If this doesn’t look right, choose a choose a different type.

RaftKeeper version

Release v2.0.0

How to reproduce

deploy a cluster of 5 raft node. using <internal_port> under tag .

Erorr:

<Error> ConfigReloader: Error loading config from 'conf/config.xml': 
Poco::Exception. Code: 1000, 
e.code() = 0, e.displayText() = 
SAXParseException: Tag mismatch in 'conf/config.xml', line 127 column 37, 
Stack trace (when copying this message, always include the lines below):

Expected behavior

run properly

[Enhancement] issue template is needed

We need an issue template which can simplify operations for contributors.

1 Issue: RaftKeeper Bug report

Problems and issues with code of RaftKeeper. If this doesn’t look right, choose a different type.

Fields marked with an asterisk (*) are required.

title should start with [Bug]

Thank you very much for submitting feedback to RaftKeeper to help develop better.

Search before asking *

Please make sure to search in the issues first to see whether the same issue was reported already.

[ ] I had searched in the issues and found no similar issues.

Version *

What is the current version

Input box: Please provide the version which you are using. The format like "version + commit id" is welcomed.

What's the problem? *

Describe the bug.

Input box: Please describe the problem in detail. The more detailed the better.

What You Expected? *

Input box:

How to Reproduce? *

Input box: Please try to give steps to facalitate the location of the problem.

Anything Else

Input box:

Are you willing to submit PR?

We look forward to developers or users to help solve the Doris problem together. If you are willing to submit a PR to fix this problem, please tick it.

[ ] Yes I am willing to submit a PR!

Code of Conduct

The Code of Conduct helps create a safe space for everyone. We require that everyone agrees to it.

[ ] I agree to follow this project's Code of Conduct *

Thanks for contributing to RaftKeeper!

2 Issue: RaftKeeper enhancement

Add an enhanment for RaftKeeper. If this doesn’t look right, choose a different type.

Fields marked with an asterisk (*) are required.

title should start with [Enhancement]

Thank you very much for submitting feedback to RaftKeeper to help develop better.

Search before asking

Please make sure to search in the issues first to see whether the same issue was reported already.

[ ] I had searched in the issues and found no similar issues. *

Description

Describe the enhancement you want, including the motivation.

Input box:

Solution

Add an overview of solution. And add some references if exist.

Input box:

Are you willing to submit PR?

We look forward to developers or users to help solve the Doris problem together. If you are willing to submit a PR to fix this problem, please tick it.

[ ] Yes I am willing to submit a PR!

Code of Conduct

The Code of Conduct helps create a safe space for everyone. We require that everyone agrees to it.

[ ] I agree to follow this project's Code of Conduct *

Thanks for contributing to RaftKeeper!

3 Issue: RaftKeeper feature request

Suggest an idea for RaftKeeper. If this doesn’t look right, choose a different type.

Fields marked with an asterisk (*) are required.

title should start with [Feature]

Thank you very much for submitting feedback to RaftKeeper to help develop better.

Search before asking

Please make sure to search in the issues first to see whether the same issue was reported already.

[ ] I had searched in the issues and found no similar issues. *

Description

Describe your ideas and needs.

Input box:

Use Case

What problem does this feature mainly solve.

Input box:

Related Issue

Is there currently another issue associated with this?

Input box:

Are you willing to submit PR?

We look forward to developers or users to help solve the Doris problem together. If you are willing to submit a PR to fix this problem, please tick it.

[ ] Yes I am willing to submit a PR!

Code of Conduct

The Code of Conduct helps create a safe space for everyone. We require that everyone agrees to it.

[ ] I agree to follow this project's Code of Conduct *

Thanks for contributing to RaftKeeper!

Fields marked with an asterisk (*) are required.

Add git commit id

Description

mntr should print git commit id
RaftKeeper log should print git commit id info.

Are you willing to submit PR?

Is RaftKeeper just taken from HouseKeeper?

The source code structure is really close, check https://github.com/ClickHouse/ClickHouse/tree/master/src/Coordination.

`numChildren` become a negative number

RaftKeeper version

raftkeeper:1.0.4

Description

stat /ckpub214.olap.jd.com/tables/sr_data_ofl_mix/app_sr_rec_other_expo_detail/05/replicas/00
cZxid = 0xcbef515
ctime = Thu Oct 19 19:05:24 CST 2023
mZxid = 0xcbef515
mtime = Thu Oct 19 19:05:24 CST 2023
pZxid = 0x27c63e9fb
cversion = 287
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x0
dataLength = 0
numChildren = -1

How to reproduce

Remove `non session_consistent` mode

Description
RaftKeeper ensure read-write consistency in one session. Now we can remove non session_consistent mode.

/cc @nicelulu

`keeper.thread_count` and `nuraft_thread_size` should take CPU core size into consideration.

Description
Now keeper.thread_count and nuraft_thread_size default value are fixed, they should be set according the CPR core size.

Also you should take vartual machine into consideration.

Add name to NIO threads

Description

It is usefull to debug.

Are you willing to submit PR?

Use relative time instead of expiration time when forwarding all local sessions

Description

Avoid problems with clock inconsistencies between servers.

Please provide a document on migrating from Zookeeper to RaftKeeper.

Is the same as clickhouse mainsite doc?

`NuRaft` shows log with unexpacted level.

Version *
v2.0.0

Describe the bug *
NuRaft shows log with unexpacted level.

NuRaft logger wraps Poco logger who has different log level settings.

NuRaft log leve

https://github.com/eBay/NuRaft/blob/2c2d2c23447701517677de9bfd3aeaa896ee7cd4/src/tracer.hxx#L54-L59
#define L_TRACE (6)
#define L_DEBUG (5)
#define L_INFO (4)
#define L_WARN (3)
#define L_ERROR (2)
#define L_FATAL (1)

Poco log level

https://github.com/pocoproject/poco/blob/57a531573fda12fb242e2149caa3f02da13ecb25/Foundation/include/Poco/Message.h#L50-L57

	PRIO_FATAL = 1,   /// A fatal error. The application will most likely terminate. This is the highest priority.
	PRIO_CRITICAL,    /// A critical error. The application might not be able to continue running successfully.
	PRIO_ERROR,       /// An error. An operation did not complete successfully, but the application as a whole is not affected.
	PRIO_WARNING,     /// A warning. An operation completed with an unexpected result.
	PRIO_NOTICE,      /// A notice, which is an information with just a higher priority.
	PRIO_INFORMATION, /// An informational message, usually denoting the successful completion of an operation.
	PRIO_DEBUG,       /// A debugging message.
	PRIO_TRACE        /// A tracing message. This is the lowest priority.

We should translate PNuRaft log level to oco log level.

Data race error

RaftKeeper version

2.0.3

Description

2023-10-23T14:45:12.7914259Z ==================
2023-10-23T14:45:12.7914449Z WARNING: ThreadSanitizer: data race (pid=8)
2023-10-23T14:45:12.7914787Z   Write of size 8 at 0x7b54000010f8 by thread T84 (mutexes: write M0):
2023-10-23T14:45:12.7917575Z     #0 std::__1::pair<std::__1::__hash_iterator<std::__1::__hash_node<RK::RequestId, void*>*>, bool> std::__1::__hash_table<RK::RequestId, RK::RequestId::RequestIdHash, std::__1::equal_to<RK::RequestId>, std::__1::allocator<RK::RequestId> >::__emplace_unique_key_args<RK::RequestId, RK::RequestId&>(RK::RequestId const&, RK::RequestId&) /home/runner/work/RaftKeeper/RaftKeeper/contrib/libcxx/include/__hash_table:2121:9 (raftkeeper+0x999bf6) (BuildId: 31ca2acb5a64ff98)
2023-10-23T14:45:12.7920163Z     #1 std::__1::pair<std::__1::__hash_iterator<std::__1::__hash_node<RK::RequestId, void*>*>, bool> std::__1::__hash_table<RK::RequestId, RK::RequestId::RequestIdHash, std::__1::equal_to<RK::RequestId>, std::__1::allocator<RK::RequestId> >::__emplace_unique_extract_key<RK::RequestId&>(RK::RequestId&, std::__1::__extract_key_self_tag) /home/runner/work/RaftKeeper/RaftKeeper/contrib/libcxx/include/__hash_table:1109:14 (raftkeeper+0x996b7b) (BuildId: 31ca2acb5a64ff98)
2023-10-23T14:45:12.7922305Z     #2 std::__1::pair<std::__1::__hash_iterator<std::__1::__hash_node<RK::RequestId, void*>*>, bool> std::__1::__hash_table<RK::RequestId, RK::RequestId::RequestIdHash, std::__1::equal_to<RK::RequestId>, std::__1::allocator<RK::RequestId> >::__emplace_unique<RK::RequestId&>(RK::RequestId&) /home/runner/work/RaftKeeper/RaftKeeper/contrib/libcxx/include/__hash_table:1079:14 (raftkeeper+0x996b7b)
2023-10-23T14:45:12.7924507Z     #3 std::__1::pair<std::__1::__hash_const_iterator<std::__1::__hash_node<RK::RequestId, void*>*>, bool> std::__1::unordered_set<RK::RequestId, RK::RequestId::RequestIdHash, std::__1::equal_to<RK::RequestId>, std::__1::allocator<RK::RequestId> >::emplace<RK::RequestId&>(RK::RequestId&) /home/runner/work/RaftKeeper/RaftKeeper/contrib/libcxx/include/unordered_set:563:30 (raftkeeper+0x996b7b)
2023-10-23T14:45:12.7925856Z     #4 RK::RequestProcessor::onError(bool, nuraft::cmd_result_code, long, int, Coordination::OpNum) /home/runner/work/RaftKeeper/RaftKeeper/src/Service/RequestProcessor.cpp:480:31 (raftkeeper+0x996b7b)
2023-10-23T14:45:12.7928221Z     #5 RK::RequestAccumulator::waitResultAndHandleError(std::__1::shared_ptr<nuraft::cmd_result<std::__1::shared_ptr<nuraft::buffer>, std::__1::shared_ptr<std::exception> > >, std::__1::vector<RK::RequestForSession, std::__1::allocator<RK::RequestForSession> > const&) /home/runner/work/RaftKeeper/RaftKeeper/src/Service/RequestAccumulator.cpp:81:32 (raftkeeper+0x97bd05) (BuildId: 31ca2acb5a64ff98)
2023-10-23T14:45:12.7929406Z     #6 RK::RequestAccumulator::run(unsigned long) /home/runner/work/RaftKeeper/RaftKeeper/src/Service/RequestAccumulator.cpp:37:17 (raftkeeper+0x97b450) (BuildId: 31ca2acb5a64ff98)
2023-10-23T14:45:12.7931400Z     #7 RK::RequestAccumulator::initialize(unsigned long, std::__1::shared_ptr<RK::KeeperDispatcher>, std::__1::shared_ptr<RK::KeeperServer>, unsigned long, unsigned long)::$_0::operator()() const /home/runner/work/RaftKeeper/RaftKeeper/src/Service/RequestAccumulator.cpp:129:49 (raftkeeper+0x97c932) (BuildId: 31ca2acb5a64ff98)
2023-10-23T14:45:12.7935236Z     #8 decltype(std::__1::forward<RK::RequestAccumulator::initialize(unsigned long, std::__1::shared_ptr<RK::KeeperDispatcher>, std::__1::shared_ptr<RK::KeeperServer>, unsigned long, unsigned long)::$_0&>(fp)()) std::__1::__invoke<RK::RequestAccumulator::initialize(unsigned long, std::__1::shared_ptr<RK::KeeperDispatcher>, std::__1::shared_ptr<RK::KeeperServer>, unsigned long, unsigned long)::$_0&>(RK::RequestAccumulator::initialize(unsigned long, std::__1::shared_ptr<RK::KeeperDispatcher>, std::__1::shared_ptr<RK::KeeperServer>, unsigned long, unsigned long)::$_0&) /home/runner/work/RaftKeeper/RaftKeeper/contrib/libcxx/include/type_traits:3676:1 (raftkeeper+0x97c932)
2023-10-23T14:45:12.7938323Z     #9 void std::__1::__invoke_void_return_wrapper<void>::__call<RK::RequestAccumulator::initialize(unsigned long, std::__1::shared_ptr<RK::KeeperDispatcher>, std::__1::shared_ptr<RK::KeeperServer>, unsigned long, unsigned long)::$_0&>(RK::RequestAccumulator::initialize(unsigned long, std::__1::shared_ptr<RK::KeeperDispatcher>, std::__1::shared_ptr<RK::KeeperServer>, unsigned long, unsigned long)::$_0&) /home/runner/work/RaftKeeper/RaftKeeper/contrib/libcxx/include/__functional_base:348:9 (raftkeeper+0x97c932)
2023-10-23T14:45:12.7940319Z     #10 std::__1::__function::__default_alloc_func<RK::RequestAccumulator::initialize(unsigned long, std::__1::shared_ptr<RK::KeeperDispatcher>, std::__1::shared_ptr<RK::KeeperServer>, unsigned long, unsigned long)::$_0, void ()>::operator()() /home/runner/work/RaftKeeper/RaftKeeper/contrib/libcxx/include/functional:1608:12 (raftkeeper+0x97c932)
2023-10-23T14:45:12.7942787Z     #11 void std::__1::__function::__policy_invoker<void ()>::__call_impl<std::__1::__function::__default_alloc_func<RK::RequestAccumulator::initialize(unsigned long, std::__1::shared_ptr<RK::KeeperDispatcher>, std::__1::shared_ptr<RK::KeeperServer>, unsigned long, unsigned long)::$_0, void ()> >(std::__1::__function::__policy_storage const*) /home/runner/work/RaftKeeper/RaftKeeper/contrib/libcxx/include/functional:2089:16 (raftkeeper+0x97c932)
2023-10-23T14:45:12.7944106Z     #12 std::__1::__function::__policy_func<void ()>::operator()() const /home/runner/work/RaftKeeper/RaftKeeper/contrib/libcxx/include/functional:2221:16 (raftkeeper+0x85cf7f) (BuildId: 31ca2acb5a64ff98)
2023-10-23T14:45:12.7947952Z     #13 std::__1::function<void ()>::operator()() const /home/runner/work/RaftKeeper/RaftKeeper/contrib/libcxx/include/functional:2560:12 (raftkeeper+0x85cf7f)
2023-10-23T14:45:12.7949444Z     #14 ThreadPoolImpl<ThreadFromGlobalPool>::worker(std::__1::__list_iterator<ThreadFromGlobalPool, void*>) /home/runner/work/RaftKeeper/RaftKeeper/src/Common/ThreadPool.cpp:265:17 (raftkeeper+0x85cf7f)
2023-10-23T14:45:12.7951546Z     #15 bool ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<bool>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()::operator()() const /home/runner/work/RaftKeeper/RaftKeeper/src/Common/ThreadPool.cpp:135:73 (raftkeeper+0x8601ff) (BuildId: 31ca2acb5a64ff98)
2023-10-23T14:45:12.7953773Z     #16 decltype(std::__1::forward<bool>(fp)()) std::__1::__invoke_constexpr<bool ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<bool>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()&>(bool&&) /home/runner/work/RaftKeeper/RaftKeeper/contrib/libcxx/include/type_traits:3682:1 (raftkeeper+0x8601ff)
2023-10-23T14:45:12.7958858Z     #17 decltype(auto) std::__1::__apply_tuple_impl<bool ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<bool>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()&, std::__1::tuple<>&>(bool&&, std::__1::tuple<>&, std::__1::__tuple_indices<>) /home/runner/work/RaftKeeper/RaftKeeper/contrib/libcxx/include/tuple:1415:1 (raftkeeper+0x8601ff)
2023-10-23T14:45:12.7961073Z     #18 decltype(auto) std::__1::apply<bool ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<bool>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()&, std::__1::tuple<>&>(bool&&, std::__1::tuple<>&) /home/runner/work/RaftKeeper/RaftKeeper/contrib/libcxx/include/tuple:1424:1 (raftkeeper+0x8601ff)
2023-10-23T14:45:12.7963192Z     #19 ThreadFromGlobalPool::ThreadFromGlobalPool<bool ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<bool>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()>(bool&&)::'lambda'()::operator()() /home/runner/work/RaftKeeper/RaftKeeper/src/Common/ThreadPool.h:178:13 (raftkeeper+0x8601ff)
2023-10-23T14:45:12.7965743Z     #20 decltype(std::__1::forward<bool>(fp)()) std::__1::__invoke<ThreadFromGlobalPool::ThreadFromGlobalPool<bool ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<bool>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()>(bool&&)::'lambda'()&>(bool&&) /home/runner/work/RaftKeeper/RaftKeeper/contrib/libcxx/include/type_traits:3676:1 (raftkeeper+0x8601ff)
2023-10-23T14:45:12.7970031Z     #21 void std::__1::__invoke_void_return_wrapper<void>::__call<ThreadFromGlobalPool::ThreadFromGlobalPool<bool ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<bool>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()>(bool&&)::'lambda'()&>(ThreadFromGlobalPool::ThreadFromGlobalPool<bool ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<bool>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()>(bool&&)::'lambda'()&) /home/runner/work/RaftKeeper/RaftKeeper/contrib/libcxx/include/__functional_base:348:9 (raftkeeper+0x860181) (BuildId: 31ca2acb5a64ff98)
2023-10-23T14:45:12.7972572Z     #22 std::__1::__function::__default_alloc_func<ThreadFromGlobalPool::ThreadFromGlobalPool<bool ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<bool>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()>(bool&&)::'lambda'(), void ()>::operator()() /home/runner/work/RaftKeeper/RaftKeeper/contrib/libcxx/include/functional:1608:12 (raftkeeper+0x860181)
2023-10-23T14:45:12.7975679Z     #23 void std::__1::__function::__policy_invoker<void ()>::__call_impl<std::__1::__function::__default_alloc_func<ThreadFromGlobalPool::ThreadFromGlobalPool<bool ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<bool>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()>(bool&&)::'lambda'(), void ()> >(std::__1::__function::__policy_storage const*) /home/runner/work/RaftKeeper/RaftKeeper/contrib/libcxx/include/functional:2089:16 (raftkeeper+0x860181)
2023-10-23T14:45:12.7976922Z     #24 std::__1::__function::__policy_func<void ()>::operator()() const /home/runner/work/RaftKeeper/RaftKeeper/contrib/libcxx/include/functional:2221:16 (raftkeeper+0x85aa87) (BuildId: 31ca2acb5a64ff98)
2023-10-23T14:45:12.7977966Z     #25 std::__1::function<void ()>::operator()() const /home/runner/work/RaftKeeper/RaftKeeper/contrib/libcxx/include/functional:2560:12 (raftkeeper+0x85aa87)
2023-10-23T14:45:12.7979107Z     #26 ThreadPoolImpl<std::__1::thread>::worker(std::__1::__list_iterator<std::__1::thread, void*>) /home/runner/work/RaftKeeper/RaftKeeper/src/Common/ThreadPool.cpp:265:17 (raftkeeper+0x85aa87)
2023-10-23T14:45:12.7981053Z     #27 void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()::operator()() const /home/runner/work/RaftKeeper/RaftKeeper/src/Common/ThreadPool.cpp:135:73 (raftkeeper+0x85db90) (BuildId: 31ca2acb5a64ff98)
2023-10-23T14:45:12.7983101Z     #28 decltype(std::__1::forward<void>(fp)()) std::__1::__invoke<void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()>(void&&) /home/runner/work/RaftKeeper/RaftKeeper/contrib/libcxx/include/type_traits:3676:1 (raftkeeper+0x85db90)
2023-10-23T14:45:12.7986770Z     #29 void std::__1::__thread_execute<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()>(std::__1::tuple<void, void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()>&, std::__1::__tuple_indices<>) /home/runner/work/RaftKeeper/RaftKeeper/contrib/libcxx/include/thread:280:5 (raftkeeper+0x85db90)
2023-10-23T14:45:12.7989301Z     #30 void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()> >(void*) /home/runner/work/RaftKeeper/RaftKeeper/contrib/libcxx/include/thread:291:5 (raftkeeper+0x85db90)
2023-10-23T14:45:12.7989318Z 
2023-10-23T14:45:12.7989579Z   Previous read of size 8 at 0x7b54000010f8 by thread T10:
2023-10-23T14:45:12.7991400Z     #0 std::__1::__hash_table<RK::RequestId, RK::RequestId::RequestIdHash, std::__1::equal_to<RK::RequestId>, std::__1::allocator<RK::RequestId> >::size() const /home/runner/work/RaftKeeper/RaftKeeper/contrib/libcxx/include/__hash_table:969:53 (raftkeeper+0x98ebfb) (BuildId: 31ca2acb5a64ff98)
2023-10-23T14:45:12.7992937Z     #1 std::__1::unordered_set<RK::RequestId, RK::RequestId::RequestIdHash, std::__1::equal_to<RK::RequestId>, std::__1::allocator<RK::RequestId> >::size() const /home/runner/work/RaftKeeper/RaftKeeper/contrib/libcxx/include/unordered_set:542:56 (raftkeeper+0x98ebfb)
2023-10-23T14:45:12.7993795Z     #2 RK::RequestProcessor::run() /home/runner/work/RaftKeeper/RaftKeeper/src/Service/RequestProcessor.cpp:53:59 (raftkeeper+0x98ebfb)
2023-10-23T14:45:12.7995629Z     #3 RK::RequestProcessor::initialize(unsigned long, std::__1::shared_ptr<RK::KeeperServer>, std::__1::shared_ptr<RK::KeeperDispatcher>, unsigned long)::$_4::operator()() const /home/runner/work/RaftKeeper/RaftKeeper/src/Service/RequestProcessor.cpp:502:49 (raftkeeper+0x99749a) (BuildId: 31ca2acb5a64ff98)
2023-10-23T14:45:12.7999205Z     #4 decltype(std::__1::forward<RK::RequestProcessor::initialize(unsigned long, std::__1::shared_ptr<RK::KeeperServer>, std::__1::shared_ptr<RK::KeeperDispatcher>, unsigned long)::$_4&>(fp)()) std::__1::__invoke_constexpr<RK::RequestProcessor::initialize(unsigned long, std::__1::shared_ptr<RK::KeeperServer>, std::__1::shared_ptr<RK::KeeperDispatcher>, unsigned long)::$_4&>(RK::RequestProcessor::initialize(unsigned long, std::__1::shared_ptr<RK::KeeperServer>, std::__1::shared_ptr<RK::KeeperDispatcher>, unsigned long)::$_4&) /home/runner/work/RaftKeeper/RaftKeeper/contrib/libcxx/include/type_traits:3682:1 (raftkeeper+0x99749a)
2023-10-23T14:45:12.8002202Z     #5 decltype(auto) std::__1::__apply_tuple_impl<RK::RequestProcessor::initialize(unsigned long, std::__1::shared_ptr<RK::KeeperServer>, std::__1::shared_ptr<RK::KeeperDispatcher>, unsigned long)::$_4&, std::__1::tuple<>&>(RK::RequestProcessor::initialize(unsigned long, std::__1::shared_ptr<RK::KeeperServer>, std::__1::shared_ptr<RK::KeeperDispatcher>, unsigned long)::$_4&, std::__1::tuple<>&, std::__1::__tuple_indices<>) /home/runner/work/RaftKeeper/RaftKeeper/contrib/libcxx/include/tuple:1415:1 (raftkeeper+0x99749a)
2023-10-23T14:45:12.8004908Z     #6 decltype(auto) std::__1::apply<RK::RequestProcessor::initialize(unsigned long, std::__1::shared_ptr<RK::KeeperServer>, std::__1::shared_ptr<RK::KeeperDispatcher>, unsigned long)::$_4&, std::__1::tuple<>&>(RK::RequestProcessor::initialize(unsigned long, std::__1::shared_ptr<RK::KeeperServer>, std::__1::shared_ptr<RK::KeeperDispatcher>, unsigned long)::$_4&, std::__1::tuple<>&) /home/runner/work/RaftKeeper/RaftKeeper/contrib/libcxx/include/tuple:1424:1 (raftkeeper+0x99749a)
2023-10-23T14:45:12.8008043Z     #7 ThreadFromGlobalPool::ThreadFromGlobalPool<RK::RequestProcessor::initialize(unsigned long, std::__1::shared_ptr<RK::KeeperServer>, std::__1::shared_ptr<RK::KeeperDispatcher>, unsigned long)::$_4>(RK::RequestProcessor::initialize(unsigned long, std::__1::shared_ptr<RK::KeeperServer>, std::__1::shared_ptr<RK::KeeperDispatcher>, unsigned long)::$_4&&)::'lambda'()::operator()() /home/runner/work/RaftKeeper/RaftKeeper/src/Common/ThreadPool.h:178:13 (raftkeeper+0x99749a)
2023-10-23T14:45:12.8013453Z     #8 decltype(std::__1::forward<RK::RequestProcessor::initialize(unsigned long, std::__1::shared_ptr<RK::KeeperServer>, std::__1::shared_ptr<RK::KeeperDispatcher>, unsigned long)::$_4>(fp)()) std::__1::__invoke<ThreadFromGlobalPool::ThreadFromGlobalPool<RK::RequestProcessor::initialize(unsigned long, std::__1::shared_ptr<RK::KeeperServer>, std::__1::shared_ptr<RK::KeeperDispatcher>, unsigned long)::$_4>(RK::RequestProcessor::initialize(unsigned long, std::__1::shared_ptr<RK::KeeperServer>, std::__1::shared_ptr<RK::KeeperDispatcher>, unsigned long)::$_4&&)::'lambda'()&>(RK::RequestProcessor::initialize(unsigned long, std::__1::shared_ptr<RK::KeeperServer>, std::__1::shared_ptr<RK::KeeperDispatcher>, unsigned long)::$_4&&) /home/runner/work/RaftKeeper/RaftKeeper/contrib/libcxx/include/type_traits:3676:1 (raftkeeper+0x99749a)
2023-10-23T14:45:12.8019429Z     #9 void std::__1::__invoke_void_return_wrapper<void>::__call<ThreadFromGlobalPool::ThreadFromGlobalPool<RK::RequestProcessor::initialize(unsigned long, std::__1::shared_ptr<RK::KeeperServer>, std::__1::shared_ptr<RK::KeeperDispatcher>, unsigned long)::$_4>(RK::RequestProcessor::initialize(unsigned long, std::__1::shared_ptr<RK::KeeperServer>, std::__1::shared_ptr<RK::KeeperDispatcher>, unsigned long)::$_4&&)::'lambda'()&>(ThreadFromGlobalPool::ThreadFromGlobalPool<RK::RequestProcessor::initialize(unsigned long, std::__1::shared_ptr<RK::KeeperServer>, std::__1::shared_ptr<RK::KeeperDispatcher>, unsigned long)::$_4>(RK::RequestProcessor::initialize(unsigned long, std::__1::shared_ptr<RK::KeeperServer>, std::__1::shared_ptr<RK::KeeperDispatcher>, unsigned long)::$_4&&)::'lambda'()&) /home/runner/work/RaftKeeper/RaftKeeper/contrib/libcxx/include/__functional_base:348:9 (raftkeeper+0x99749a)
2023-10-23T14:45:12.8023019Z     #10 std::__1::__function::__default_alloc_func<ThreadFromGlobalPool::ThreadFromGlobalPool<RK::RequestProcessor::initialize(unsigned long, std::__1::shared_ptr<RK::KeeperServer>, std::__1::shared_ptr<RK::KeeperDispatcher>, unsigned long)::$_4>(RK::RequestProcessor::initialize(unsigned long, std::__1::shared_ptr<RK::KeeperServer>, std::__1::shared_ptr<RK::KeeperDispatcher>, unsigned long)::$_4&&)::'lambda'(), void ()>::operator()() /home/runner/work/RaftKeeper/RaftKeeper/contrib/libcxx/include/functional:1608:12 (raftkeeper+0x99749a)
2023-10-23T14:45:12.8027295Z     #11 void std::__1::__function::__policy_invoker<void ()>::__call_impl<std::__1::__function::__default_alloc_func<ThreadFromGlobalPool::ThreadFromGlobalPool<RK::RequestProcessor::initialize(unsigned long, std::__1::shared_ptr<RK::KeeperServer>, std::__1::shared_ptr<RK::KeeperDispatcher>, unsigned long)::$_4>(RK::RequestProcessor::initialize(unsigned long, std::__1::shared_ptr<RK::KeeperServer>, std::__1::shared_ptr<RK::KeeperDispatcher>, unsigned long)::$_4&&)::'lambda'(), void ()> >(std::__1::__function::__policy_storage const*) /home/runner/work/RaftKeeper/RaftKeeper/contrib/libcxx/include/functional:2089:16 (raftkeeper+0x99749a)
2023-10-23T14:45:12.8028657Z     #12 std::__1::__function::__policy_func<void ()>::operator()() const /home/runner/work/RaftKeeper/RaftKeeper/contrib/libcxx/include/functional:2221:16 (raftkeeper+0x85aa87) (BuildId: 31ca2acb5a64ff98)
2023-10-23T14:45:12.8029611Z     #13 std::__1::function<void ()>::operator()() const /home/runner/work/RaftKeeper/RaftKeeper/contrib/libcxx/include/functional:2560:12 (raftkeeper+0x85aa87)
2023-10-23T14:45:12.8030699Z     #14 ThreadPoolImpl<std::__1::thread>::worker(std::__1::__list_iterator<std::__1::thread, void*>) /home/runner/work/RaftKeeper/RaftKeeper/src/Common/ThreadPool.cpp:265:17 (raftkeeper+0x85aa87)
2023-10-23T14:45:12.8032635Z     #15 void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()::operator()() const /home/runner/work/RaftKeeper/RaftKeeper/src/Common/ThreadPool.cpp:135:73 (raftkeeper+0x85db90) (BuildId: 31ca2acb5a64ff98)
2023-10-23T14:45:12.8034685Z     #16 decltype(std::__1::forward<void>(fp)()) std::__1::__invoke<void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()>(void&&) /home/runner/work/RaftKeeper/RaftKeeper/contrib/libcxx/include/type_traits:3676:1 (raftkeeper+0x85db90)
2023-10-23T14:45:12.8038204Z     #17 void std::__1::__thread_execute<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()>(std::__1::tuple<void, void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()>&, std::__1::__tuple_indices<>) /home/runner/work/RaftKeeper/RaftKeeper/contrib/libcxx/include/thread:280:5 (raftkeeper+0x85db90)
2023-10-23T14:45:12.8040849Z     #18 void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()> >(void*) /home/runner/work/RaftKeeper/RaftKeeper/contrib/libcxx/include/thread:291:5 (raftkeeper+0x85db90)

How to reproduce

Found in CI test here

Organize the integration tests log output

Description
Now integration tests log output is messy. We'd better do some optimization.

We can leverage the python logging component and setup log level carfully.
We should print raftkeeper instances log if a test failed.

Optimize clang-format template

If this doesn’t look right, choose a different type.

Description

regroup and order includes
fix some invalid values
standard: c++11 -> c++20
break braces in lambda functions
column limit 140 -> 120

Is there any information available on RaftKeeper

is there any introduction documents about RaftKeeper, such as cross-site active-active deployment or how to solve the problem that ZooKeeper cannot break through high-throughput writing and low-latency data replication in ClickHouse? thanks

chore: code style

If this doesn’t look right, choose a different type.

Description

code style fix

Has anyone compared with ClickHouse Keeper in performance?

I spent a day doing a quick read of the code today, and find that this project looks like it's disigned and implemented by experienced developers. Since there's a lot of object-oriented programming and clever optimization. I hope to introduce this project into our products. But still I have a question about whether it can be far ahead of clickhouse keeper in term of performance. Can friends offer some experience? Thanks in advance :)

Support Persistent and PersistentRecursive watcher, add AddWatch and RemoveWatches

Description

Accomplish tasks on #35.

Are you willing to submit PR?

yes.

Check-style does not match the result of clang-format

RaftKeeper version

Description

run clang-format -i src/Servcie/Settings.cpp will format Settings.cpp by rules in .clang-format:

 void Settings::dump(WriteBufferFromOwnString & buf) const
 {
-    auto write_int = [&buf](int64_t value)
-    {
+    auto write_int = [&buf](int64_t value) {
         writeIntText(value, buf);
         buf.write('\n');
     };

This will not pass check-style, error msg:

Error: yle [Error] **********************************************************************************************************
/home/runner/work/RaftKeeper/RaftKeeper/src/Service/Settings.cpp:140: auto write_int = [&buf](int64_t value) {

How to reproduce

clang-format -i src/Service/Settings.cpp

Add some limitations to protect system.

Description
Add some limitations to protect system, such as max_node_child_size, max_node_deepth etc.

Solution

Make the limitations confiurable. Below are some limitations:

max_node_child_size : max child size for a node
max_node_deepth: max deepth of data tree
max_node_name_length: max node name length
max_node_value_length: max node value length

max_connections

code style output not expected

Problems and issues with code of RaftKeeper. If this doesn’t look right, choose a choose a different type.

discription

see yle pos

Error: yle [Error] **********************************************************************************************************

/home/runner/work/RaftKeeper/RaftKeeper/src/Service/Settings.cpp:123:    auto write_int = [&buf](int64_t value) {

/home/runner/work/RaftKeeper/RaftKeeper/src/Service/KeeperStore.cpp:1664:    auto write_int_vec = [&buf](const std::vector<int64_t> & session_ids) {

/home/runner/work/RaftKeeper/RaftKeeper/src/Service/KeeperStore.cpp:1687:    auto write_str_set = [&buf](const std::unordered_set<String> & ephemeral_paths) {

/home/runner/work/RaftKeeper/RaftKeeper/src/Service/NuRaftStateManager.cpp:160:    std::for_each(ret_cluster_config->get_servers().cbegin(), ret_cluster_config->get_servers().cend(), [&s](ptr<srv_config> srv) {

/home/runner/work/RaftKeeper/RaftKeeper/src/Service/tests/gtest_raft_snapshot.cpp:274:        map.forEach([&ano_map](const auto & key, const auto & value) {

/home/runner/work/RaftKeeper/RaftKeeper/src/Service/tests/gtest_raft_snapshot.cpp:302:    auto filter_auth = [](KeeperStore::SessionAndAuth & auth_ids) {

/home/runner/work/RaftKeeper/RaftKeeper/src/Service/tests/gtest_raft_snapshot.cpp:703:    cmd_result<bool>::handler_type handler = [log, &snapshot_done, &mutex, &cv](bool, ptr<std::exception> &) {


Error: pos [Error] **********************************************************************************************************

/home/runner/work/RaftKeeper/RaftKeeper/src/Service/NuRaftLogSnapshot.h:16[9](https://github.com/JDRaftKeeper/RaftKeeper/actions/runs/4571187081/jobs/8069249529?pr=28#step:4:10): ore ==> or

The CPU usage is very high occasionally.

RaftKeeper version

v2.0.2

Description

Occasionally, the CPU usage is very high.

How to reproduce

Actually it is difficult to reproduce, but if it runs for a long time, it may occur.

Forward new session and update session request with forwarder

If this doesn’t look right, choose a different type.

Description

The auto-forwarding feature of NuRaft is not stable. We should forward new session and update session request with forwarder. So we can disable auto-forwarding feature of NuRaft.

RaftKeeper cluster configuration on a single machine

System: MacOS Ventura 13.2.1
Build Version: the lastest commit bf5f6c1

I built RaftKeeper on my machine following the instruction in README. Then I modified the configuration file conf/config.xml to set up a 3 nodes Raft cluster:

<?xml version="1.0"?>
<raftkeeper>
    <!-- System timezone, default is empty, will use OS default timezone. -->
    <!-- <timezone></timezone>-->
    <logger>
        <!-- Valid values are: fatal, error, warning, information, debug, trace -->
        <!-- <level>information</level> -->
        <!-- Logger file path, default is empty string. If is empty string will not log to file -->
        <path>./log/raftkeeper.log</path>
        <!-- Error logger file path, default is empty string. If is empty string will not log to file -->
        <err_log_path>./log/raftkeeper.err.log</err_log_path>
        <!-- Log file size limit, when reached will create another file, default is 100M. -->
        <!-- <size>100M</size> -->
        <!-- How many log file to keep, default is 10. -->
        <!-- <count>10</count> -->
        <!-- Whether compress log, default is true. -->
        <!-- <compress>true</compress> -->
        <!-- Whether print log to console, default is false-->
        <!-- <log_to_console>false</log_to_console> -->
    </logger>

    <!-- <core_dump> -->
        <!-- 1 GiB by default. If more - it writes to disk too long. -->
        <!-- <size_limit>1073741824</size_limit> -->
    <!-- </core_dump> -->

    <keeper>
        <!-- My id in cluster. -->
        <my_id>1</my_id>

        <!-- Binging host, default is "0.0.0.0." -->
        <!-- <host>0.0.0.0</host> -->

        <!-- Port for user request and 4lw command -->
        <!-- <port>8101</port> -->

        <!-- Port for follower forward write request and session info to leader. -->
        <!-- <forwarding_port>8102</forwarding_port> -->

        <!-- Port for Raft internal usage: heartbeat, log replicate, leader selection etc. -->
        <!-- <internal_port>8103</internal_port> -->

        <!-- Raft log store directory -->
        <log_dir>./data/log</log_dir>

        <!-- Raft snapshot store directory -->
        <snapshot_dir>./data/snapshot</snapshot_dir>

        <!-- Max snapshot interval in second. -->
        <!-- <snapshot_create_interval>3600</snapshot_create_interval> -->

        <!-- Processor thread count, default is 16. -->
        <!-- <thread_count>16</thread_count> -->

        <!-- 4lwd command white list, default "conf,cons,crst,envi,ruok,srst,srvr,stat,wchs,dirs,mntr,isro,lgif,rqld" -->
        <!-- <four_letter_word_white_list></four_letter_word_white_list> -->

        <!-- Super digest for root user, default is empty string.
            See https://zookeeper.apache.org/doc/r3.5.2-alpha/zookeeperAdmin.html  -->
        <!-- <super_digest></super_digest> -->

        <!-- NuRaft related settings -->
        <raft_settings>
            <!-- User session timeout in millisecond, default is 30000. -->
            <!-- <session_timeout_ms>30000</session_timeout_ms> -->

            <!-- User operation timeout in millisecond, default is 20000. -->
            <!-- <operation_timeout_ms>20000</operation_timeout_ms> -->

            <!-- Leader will check whether session is dead in this period, default is 1000. -->
            <!-- <dead_session_check_period_ms>100</dead_session_check_period_ms> -->

            <!-- NuRaft heart beat interval in millisecond, default is 500. -->
            <!-- <heart_beat_interval_ms>500</heart_beat_interval_ms> -->

            <!-- NuRaft election timeout lower bound, default is 10s. -->
            <!-- <election_timeout_lower_bound_ms>10000</election_timeout_lower_bound_ms> -->

            <!-- NuRaft election timeout upper bound, default is 20s. -->
            <!-- <election_timeout_upper_bound_ms>20000</election_timeout_upper_bound_ms> -->

            <!-- Keeper will keep at least this log item, default 1_000_000. -->
            <!-- <reserved_log_items>1000000</reserved_log_items> -->

            <!-- Create snapshot in this log size, default is 3_000_000. -->
            <!-- <snapshot_distance>3000000</snapshot_distance> -->

            <!-- How many snapshot to keep, default is 5. -->
            <!-- <max_stored_snapshots>5</max_stored_snapshots> -->

            <!-- Startup time in millisecond, default is 6000000ms. Because will load data, should set to a big value. -->
            <!-- <startup_timeout>6000000</startup_timeout> -->

            <!-- Shutdown time in millisecond, default is 5000ms. -->
            <!-- <shutdown_timeout>5000</shutdown_timeout> -->

            <!-- NuRaft log level, default is information. -->
            <!-- <raft_logs_level>information</raft_logs_level> -->

            <!-- NuRaft thread pool size, default is 32. -->
            <!-- <nuraft_thread_size>32</nuraft_thread_size> -->

            <!-- Log gap (compared to the leader's latest log) for treating this node as fresh, default is 200. -->
            <!-- <fresh_log_gap>200</fresh_log_gap> -->

            <!-- Auto update configuration retry times, default is 30. -->
            <!-- <configuration_change_tries_count>30</configuration_change_tries_count> -->

            <!-- NuRaft append entries max batch size, default is 1000. -->
            <!-- <max_batch_size>1000</max_batch_size> -->

            <!-- Raft log fsync mode:
                    fsync_parallel : The leader can do log replication and log persisting in parallel,
                        thus it can reduce the latency of write operation path. In this mode data is safety.
                    fsync : The leader and follower do log persisting synchronously. In this mode data is safety.
                    fsync_batch : The leader and follower do log persisting asynchronously and in batch.
                        In this mode data is less safety.
            -->
            <!-- <log_fsync_mode>fsync_parallel</log_fsync_mode> -->

            <!-- If log_fsync_mode is fsync_batch, will fsync log after x appending entries, default value is 1000. -->
            <!-- <log_fsync_interval>1000</log_fsync_interval> -->
        </raft_settings>

        <!-- If you want a RaftKeeper cluster, you can uncomment this and configure it carefully -->
        <cluster>
            <server>
                <id>1</id>
                <host>127.0.0.1</host>
                <!-- <internal_port>8103</internal_port> -->
                <!-- <forwarding_port>8102</forwarding_port> -->
                <!-- `true` if this node is learner. Learner will not initiate or participate in leader election. -->
                <!-- <learner>false</learner> -->
                <!-- Priority of this server, default is 1 and if is 0 the server will never be leader. -->
                <!-- <priority>1</priority> -->
            </server>
            <server>
                <id>2</id>
                <host>127.0.0.1</host>
            </server>
            <server>
                <id>3</id>
                <host>127.0.0.1</host>
            </server>
        </cluster>
    </keeper>
</raftkeeper>

With that configuration and I run RaftKeeper with command ./lib/raftkeeper server --config=conf/config.xml, got the logs below:

2023.03.24 02:56:29.806859 [ 2836175 ] {} <Information> : Starting RaftKeeper v2.0.1, no build id, PID 35071
2023.03.24 02:56:29.914603 [ 2836175 ] {} <Information> LogSegmentStore: Create LogSegmentStore ./data/log.
2023.03.24 02:56:29.914748 [ 2836175 ] {} <Information> LogSegmentStore: Begin init log segment store, max log size 1048576000 bytes, max segment count 50.
2023.03.24 02:56:29.916808 [ 2836175 ] {} <Information> LogSegment: Created new segment ./data/log/log_1_open_20230324025629, seg_fd 7, first index 1
2023.03.24 02:56:29.917013 [ 2836175 ] {} <Information> FileLogStore: Init file log store, last log index 0, log dir ./data/log
2023.03.24 02:56:29.917144 [ 2836175 ] {} <Information> NuRaftStateManager: Create ForwardingConnection for 2, 127.0.0.1:8102
2023.03.24 02:56:29.917217 [ 2836175 ] {} <Information> NuRaftStateManager: Create ForwardingConnection for 2, 127.0.0.1:8102, thread 0, 0x10a154018
2023.03.24 02:56:29.917256 [ 2836175 ] {} <Information> NuRaftStateManager: Create ForwardingConnection for 2, 127.0.0.1:8102, thread 1, 0x10a1541d8
2023.03.24 02:56:29.917289 [ 2836175 ] {} <Information> NuRaftStateManager: Create ForwardingConnection for 2, 127.0.0.1:8102, thread 2, 0x10a154398
2023.03.24 02:56:29.917322 [ 2836175 ] {} <Information> NuRaftStateManager: Create ForwardingConnection for 2, 127.0.0.1:8102, thread 3, 0x10a154558
2023.03.24 02:56:29.917355 [ 2836175 ] {} <Information> NuRaftStateManager: Create ForwardingConnection for 2, 127.0.0.1:8102, thread 4, 0x10a154718
2023.03.24 02:56:29.917387 [ 2836175 ] {} <Information> NuRaftStateManager: Create ForwardingConnection for 2, 127.0.0.1:8102, thread 5, 0x10a1548d8
2023.03.24 02:56:29.917551 [ 2836175 ] {} <Information> NuRaftStateManager: Create ForwardingConnection for 2, 127.0.0.1:8102, thread 6, 0x10a154a98
2023.03.24 02:56:29.917640 [ 2836175 ] {} <Information> NuRaftStateManager: Create ForwardingConnection for 2, 127.0.0.1:8102, thread 7, 0x10a154c58
2023.03.24 02:56:29.917688 [ 2836175 ] {} <Information> NuRaftStateManager: Create ForwardingConnection for 2, 127.0.0.1:8102, thread 8, 0x10a154e18
2023.03.24 02:56:29.917772 [ 2836175 ] {} <Information> NuRaftStateManager: Create ForwardingConnection for 2, 127.0.0.1:8102, thread 9, 0x10a154fd8
2023.03.24 02:56:29.917815 [ 2836175 ] {} <Information> NuRaftStateManager: Create ForwardingConnection for 2, 127.0.0.1:8102, thread 10, 0x10a155198
2023.03.24 02:56:29.918262 [ 2836175 ] {} <Information> NuRaftStateManager: Create ForwardingConnection for 2, 127.0.0.1:8102, thread 11, 0x10a155358
2023.03.24 02:56:29.918348 [ 2836175 ] {} <Information> NuRaftStateManager: Create ForwardingConnection for 2, 127.0.0.1:8102, thread 12, 0x10a155518
2023.03.24 02:56:29.918385 [ 2836175 ] {} <Information> NuRaftStateManager: Create ForwardingConnection for 2, 127.0.0.1:8102, thread 13, 0x10a1556d8
2023.03.24 02:56:29.918417 [ 2836175 ] {} <Information> NuRaftStateManager: Create ForwardingConnection for 2, 127.0.0.1:8102, thread 14, 0x10a155898
2023.03.24 02:56:29.918449 [ 2836175 ] {} <Information> NuRaftStateManager: Create ForwardingConnection for 2, 127.0.0.1:8102, thread 15, 0x10a155a58
2023.03.24 02:56:29.918537 [ 2836175 ] {} <Information> NuRaftStateManager: Create ForwardingConnection for 3, 127.0.0.1:8102
2023.03.24 02:56:29.918583 [ 2836175 ] {} <Information> NuRaftStateManager: Create ForwardingConnection for 3, 127.0.0.1:8102, thread 0, 0x10a155c18
2023.03.24 02:56:29.918616 [ 2836175 ] {} <Information> NuRaftStateManager: Create ForwardingConnection for 3, 127.0.0.1:8102, thread 1, 0x10a155dd8
2023.03.24 02:56:29.918653 [ 2836175 ] {} <Information> NuRaftStateManager: Create ForwardingConnection for 3, 127.0.0.1:8102, thread 2, 0x10a155f98
2023.03.24 02:56:29.918687 [ 2836175 ] {} <Information> NuRaftStateManager: Create ForwardingConnection for 3, 127.0.0.1:8102, thread 3, 0x10a156158
2023.03.24 02:56:29.918719 [ 2836175 ] {} <Information> NuRaftStateManager: Create ForwardingConnection for 3, 127.0.0.1:8102, thread 4, 0x10a156318
2023.03.24 02:56:29.918751 [ 2836175 ] {} <Information> NuRaftStateManager: Create ForwardingConnection for 3, 127.0.0.1:8102, thread 5, 0x10a1564d8
2023.03.24 02:56:29.918852 [ 2836175 ] {} <Information> NuRaftStateManager: Create ForwardingConnection for 3, 127.0.0.1:8102, thread 6, 0x10a156698
2023.03.24 02:56:29.918880 [ 2836175 ] {} <Information> NuRaftStateManager: Create ForwardingConnection for 3, 127.0.0.1:8102, thread 7, 0x10a156858
2023.03.24 02:56:29.919076 [ 2836175 ] {} <Information> NuRaftStateManager: Create ForwardingConnection for 3, 127.0.0.1:8102, thread 8, 0x10a156a18
2023.03.24 02:56:29.919154 [ 2836175 ] {} <Information> NuRaftStateManager: Create ForwardingConnection for 3, 127.0.0.1:8102, thread 9, 0x10a156bd8
2023.03.24 02:56:29.919186 [ 2836175 ] {} <Information> NuRaftStateManager: Create ForwardingConnection for 3, 127.0.0.1:8102, thread 10, 0x10a156d98
2023.03.24 02:56:29.919237 [ 2836175 ] {} <Information> NuRaftStateManager: Create ForwardingConnection for 3, 127.0.0.1:8102, thread 11, 0x10a156f58
2023.03.24 02:56:29.919268 [ 2836175 ] {} <Information> NuRaftStateManager: Create ForwardingConnection for 3, 127.0.0.1:8102, thread 12, 0x10a157118
2023.03.24 02:56:29.919326 [ 2836175 ] {} <Information> NuRaftStateManager: Create ForwardingConnection for 3, 127.0.0.1:8102, thread 13, 0x10a1572d8
2023.03.24 02:56:29.919349 [ 2836175 ] {} <Information> NuRaftStateManager: Create ForwardingConnection for 3, 127.0.0.1:8102, thread 14, 0x10a157498
2023.03.24 02:56:29.919374 [ 2836175 ] {} <Information> NuRaftStateManager: Create ForwardingConnection for 3, 127.0.0.1:8102, thread 15, 0x10a157658
2023.03.24 02:56:29.919397 [ 2836175 ] {} <Information> NuRaftStateManager: raft cluster config :  127.0.0.1:8103 127.0.0.1:8103 127.0.0.1:8103
2023.03.24 02:56:29.919628 [ 2836175 ] {} <Information> KeeperStateMachine: begin init state machine, snapshot directory ./data/snapshot
2023.03.24 02:56:29.920045 [ 2836175 ] {} <Information> RaftTaskManager: Open task for task, fd 8, path ./data/snapshot/committed.task
2023.03.24 02:56:29.920127 [ 2836175 ] {} <Warning> RaftTaskManager: Read last committed index failed -1
2023.03.24 02:56:29.920322 [ 2836201 ] {} <Information> RaftTaskManager: try schedule, shut down false
2023.03.24 02:56:29.920464 [ 2836175 ] {} <Information> KeeperSnapshotManager: Skip no snapshot file committed.task
2023.03.24 02:56:29.920503 [ 2836175 ] {} <Information> KeeperSnapshotManager: Load snapshot metas 0 from snapshot directory ./data/snapshot
2023.03.24 02:56:29.920529 [ 2836175 ] {} <Information> KeeperSnapshotManager: Get last snapshot, snapshot size 0
2023.03.24 02:56:29.920551 [ 2836175 ] {} <Information> KeeperStateMachine: Load snapshot meta size 0, last log index 0 in snapshot
2023.03.24 02:56:29.920580 [ 2836175 ] {} <Information> KeeperStateMachine: Begin replay log, first log index 1 and last log index 0 in log file ( prev index 0, log index 0 )
2023.03.24 02:56:29.920719 [ 2836175 ] {} <Information> KeeperStateMachine: Apply log done, ephemeral sessions 0 nodes 0
2023.03.24 02:56:29.920881 [ 2836175 ] {} <Information> KeeperStateMachine: Replay last committed index 0 in log store
2023.03.24 02:56:29.920958 [ 2836175 ] {} <Information> KeeperStateMachine: Starting background creating snapshot thread.
2023.03.24 02:56:29.921993 [ 2836175 ] {} <Information> NuRaft: Raft ASIO listener initiated, UNSECURED
2023.03.24 02:56:29.922135 [ 2836175 ] {} <Warning> NuRaftStateManager: Raft srv_state file not exist, maybe this is the first startup.
2023.03.24 02:56:29.922204 [ 2836175 ] {} <Information> NuRaftStateManager: load config with initial cluster config.
2023.03.24 02:56:29.922232 [ 2836175 ] {} <Information> KeeperStateMachine: last_snapshot invoke
2023.03.24 02:56:29.922256 [ 2836175 ] {} <Information> KeeperSnapshotManager: Get last snapshot, snapshot size 0
2023.03.24 02:56:29.922292 [ 2836175 ] {} <Information> NuRaft: parameters: timeout 10000 - 20000, heartbeat 500, leadership expiry 10000, max batch 100, backoff 50, snapshot distance 3000000, enable randomized snapshot creation NO, log sync stop gap 99999, reserved logs 1000000, client timeout 20000, auto forwarding ON, API call type BLOCKING, custom commit quorum size 0, custom election quorum size 0, snapshot receiver INCLUDED, leadership transfer wait time 0, grace period of lagging state machine 0, snapshot IO: BLOCKING, parallel log appending: ON
2023.03.24 02:56:29.922349 [ 2836175 ] {} <Information> NuRaft: new timeout range: 10000 -- 20000
2023.03.24 02:56:29.922385 [ 2836175 ] {} <Information> NuRaft:    === INIT RAFT SERVER ===
commit index 0
term 0
election timer allowed
log store start 1, end 0
config log idx 0, prev log idx 0
2023.03.24 02:56:29.922453 [ 2836175 ] {} <Information> NuRaft: peer 1: DC ID 0, 127.0.0.1:8103, voting member, 1
peer 2: DC ID 0, 127.0.0.1:8103, voting member, 1
peer 3: DC ID 0, 127.0.0.1:8103, voting member, 1
my id: 1, voting_member
num peers: 2
2023.03.24 02:56:29.922483 [ 2836175 ] {} <Information> NuRaft: global manager does not exist. will use local thread for commit and append
2023.03.24 02:56:29.922563 [ 2836175 ] {} <Information> NuRaft: wait for HB, for 50 + [10000, 20000] ms
2023.03.24 02:56:29.922924 [ 2836236 ] {} <Information> NuRaft: bg append_entries thread initiated
2023.03.24 02:56:48.868253 [ 2836203 ] {} <Warning> NuRaft: Election timeout, initiate leader election
2023.03.24 02:56:48.868621 [ 2836203 ] {} <Information> NuRaft: [PRIORITY] decay, target 1 -> 1, mine 1
2023.03.24 02:56:48.868767 [ 2836203 ] {} <Information> NuRaft: [ELECTION TIMEOUT] current role: follower, log last term 0, state term 0, target p 1, my p 1, hb dead, pre-vote NOT done
2023.03.24 02:56:48.868882 [ 2836203 ] {} <Information> NuRaft: [PRE-VOTE INIT] my id 1, my role candidate, term 0, log idx 0, log term 0, priority (target 1 / mine 1)
2023.03.24 02:56:48.872425 [ 2836205 ] {} <Information> NuRaft: 0x108ff0f18 connected to 127.0.0.1:8103 (as a client)
2023.03.24 02:56:48.872691 [ 2836209 ] {} <Information> NuRaft: 0x108ff0d98 connected to 127.0.0.1:8103 (as a client)
2023.03.24 02:56:48.872968 [ 2836211 ] {} <Information> NuRaft: receive a incoming rpc connection
2023.03.24 02:56:48.873312 [ 2836211 ] {} <Information> NuRaft: session 1 got connection from 127.0.0.1:56195 (as a server)
2023.03.24 02:56:48.873502 [ 2836211 ] {} <Information> NuRaft: receive a incoming rpc connection
2023.03.24 02:56:48.873588 [ 2836211 ] {} <Information> NuRaft: session 2 got connection from 127.0.0.1:56194 (as a server)
2023.03.24 02:56:48.873852 [ 2836214 ] {} <Information> NuRaft: [PRE-VOTE REQ] my role candidate, from peer 1, log term: req 0 / mine 0
last idx: req 0 / mine 0, term: req 0 / mine 0
HB dead
2023.03.24 02:56:48.874075 [ 2836214 ] {} <Information> NuRaft: pre-vote decision: O (grant)
2023.03.24 02:56:48.874449 [ 2836216 ] {} <Information> NuRaft: [PRE-VOTE REQ] my role candidate, from peer 1, log term: req 0 / mine 0
last idx: req 0 / mine 0, term: req 0 / mine 0
HB dead
2023.03.24 02:56:48.874617 [ 2836216 ] {} <Information> NuRaft: pre-vote decision: O (grant)
2023.03.24 02:56:48.874854 [ 2836210 ] {} <Information> NuRaft: [PRE-VOTE RESP] peer 1 (O), term 0, resp term 0, my role candidate, dead 2, live 0, abandoned 0, num voting members 3, quorum 2
2023.03.24 02:56:48.875092 [ 2836210 ] {} <Information> NuRaft: [PRE-VOTE DONE] SUCCESS, term 0
2023.03.24 02:56:48.875158 [ 2836210 ] {} <Information> NuRaft: [PRE-VOTE DONE] initiate actual vote
2023.03.24 02:56:48.875774 [ 2836210 ] {} <Information> NuRaftStateManager: save srv_state with term 1 and vote_for 1
2023.03.24 02:56:48.875842 [ 2836210 ] {} <Information> NuRaft: [VOTE INIT] my id 1, my role candidate, term 1, log idx 0, log term 0, priority (target 1 / mine 1)
2023.03.24 02:56:48.876186 [ 2836219 ] {} <Information> NuRaft: [PRE-VOTE RESP] peer 1 (O), term 0, resp term 0, my role candidate, dead 3, live 0, abandoned 0, num voting members 3, quorum 2
2023.03.24 02:56:48.876394 [ 2836219 ] {} <Information> NuRaft: [PRE-VOTE DONE] SUCCESS, term 0
2023.03.24 02:56:48.876487 [ 2836219 ] {} <Information> NuRaft: [PRE-VOTE DONE] actual vote is already initiated, do nothing
2023.03.24 02:56:48.876777 [ 2836225 ] {} <Information> NuRaft: [VOTE REQ] my role candidate, from peer 1, log term: req 0 / mine 0
last idx: req 0 / mine 0, term: req 1 / mine 1
priority: target 1 / mine 1, voted_for 1
2023.03.24 02:56:48.877009 [ 2836225 ] {} <Information> NuRaft: decision: O (grant), voted_for 1, term 1
2023.03.24 02:56:48.877643 [ 2836225 ] {} <Information> NuRaftStateManager: save srv_state with term 1 and vote_for 1
2023.03.24 02:56:48.877916 [ 2836220 ] {} <Information> NuRaft: [VOTE REQ] my role candidate, from peer 1, log term: req 0 / mine 0
last idx: req 0 / mine 0, term: req 1 / mine 1
priority: target 1 / mine 1, voted_for 1
2023.03.24 02:56:48.878016 [ 2836220 ] {} <Information> NuRaft: decision: O (grant), voted_for 1, term 1
2023.03.24 02:56:48.878363 [ 2836220 ] {} <Information> NuRaftStateManager: save srv_state with term 1 and vote_for 1
2023.03.24 02:56:48.878579 [ 2836226 ] {} <Information> NuRaft: [VOTE RESP] peer 1 (O), resp term 1, my role candidate, granted 2, responded 2, num voting members 3, quorum 2
2023.03.24 02:56:48.878686 [ 2836226 ] {} <Information> NuRaft: Server is elected as leader for term 1
2023.03.24 02:56:48.878764 [ 2836226 ] {} <Information> NuRaft: number of pending commit elements: 0
2023.03.24 02:56:48.878801 [ 2836226 ] {} <Information> NuRaft: state machine commit index 0, precommit index 0, last log index 0
2023.03.24 02:56:48.878901 [ 2836226 ] {} <Information> NuRaft: [BECOME LEADER] appended new config at 1
2023.03.24 02:56:48.879411 [ 2836226 ] {} <Information> NuRaft:   === LEADER (term 1) ===
2023.03.24 02:56:48.879552 [ 2836228 ] {} <Information> NuRaft: Election completed, will ignore the voting result from this server
2023.03.24 02:56:48.880097 [ 2836204 ] {} <Warning> NuRaft: Receive AppendEntriesRequest from another leader (1) with same term, there must be a bug. Ignore it instead of exit.
2023.03.24 02:56:48.880311 [ 2836204 ] {} <Warning> NuRaft: no response is returned from raft message handler
2023.03.24 02:56:48.880517 [ 2836229 ] {} <Warning> NuRaft: Receive AppendEntriesRequest from another leader (1) with same term, there must be a bug. Ignore it instead of exit.
2023.03.24 02:56:48.880820 [ 2836229 ] {} <Warning> NuRaft: no response is returned from raft message handler
2023.03.24 02:56:48.881062 [ 2836203 ] {} <Warning> NuRaft: peer (2) response error: failed to read response to peer 2, 127.0.0.1:8103, error 2, End of file
2023.03.24 02:56:48.881765 [ 2836213 ] {} <Warning> NuRaft: peer (3) response error: failed to read response to peer 3, 127.0.0.1:8103, error 2, End of file
2023.03.24 02:56:48.883992 [ 2836175 ] {} <Information> NuRaftStateManager: Create ForwardingConnection for 2, 127.0.0.1:8102
2023.03.24 02:56:48.884147 [ 2836175 ] {} <Information> NuRaftStateManager: Create ForwardingConnection for 2, 127.0.0.1:8102, thread 0, 0x10a154018
2023.03.24 02:56:48.884199 [ 2836175 ] {} <Information> NuRaftStateManager: Create ForwardingConnection for 2, 127.0.0.1:8102, thread 1, 0x10a1541d8
2023.03.24 02:56:48.884232 [ 2836175 ] {} <Information> NuRaftStateManager: Create ForwardingConnection for 2, 127.0.0.1:8102, thread 2, 0x10a154398
2023.03.24 02:56:48.884260 [ 2836175 ] {} <Information> NuRaftStateManager: Create ForwardingConnection for 2, 127.0.0.1:8102, thread 3, 0x10a154558
2023.03.24 02:56:48.884531 [ 2836565 ] {} <Information> KeeperDispatcher: start session clear task
2023.03.24 02:56:48.884681 [ 2836175 ] {} <Information> NuRaftStateManager: Create ForwardingConnection for 2, 127.0.0.1:8102, thread 4, 0x10a154718
2023.03.24 02:56:48.884737 [ 2836175 ] {} <Information> NuRaftStateManager: Create ForwardingConnection for 2, 127.0.0.1:8102, thread 5, 0x10a1548d8
2023.03.24 02:56:48.884758 [ 2836175 ] {} <Information> NuRaftStateManager: Create ForwardingConnection for 2, 127.0.0.1:8102, thread 6, 0x10a154a98
2023.03.24 02:56:48.884779 [ 2836175 ] {} <Information> NuRaftStateManager: Create ForwardingConnection for 2, 127.0.0.1:8102, thread 7, 0x10a154c58
2023.03.24 02:56:48.884799 [ 2836175 ] {} <Information> NuRaftStateManager: Create ForwardingConnection for 2, 127.0.0.1:8102, thread 8, 0x10a154e18
2023.03.24 02:56:48.884847 [ 2836175 ] {} <Information> NuRaftStateManager: Create ForwardingConnection for 2, 127.0.0.1:8102, thread 9, 0x10a154fd8
2023.03.24 02:56:48.884867 [ 2836175 ] {} <Information> NuRaftStateManager: Create ForwardingConnection for 2, 127.0.0.1:8102, thread 10, 0x10a155198
2023.03.24 02:56:48.884885 [ 2836175 ] {} <Information> NuRaftStateManager: Create ForwardingConnection for 2, 127.0.0.1:8102, thread 11, 0x10a155358
2023.03.24 02:56:48.885017 [ 2836175 ] {} <Information> NuRaftStateManager: Create ForwardingConnection for 2, 127.0.0.1:8102, thread 12, 0x10a155518
2023.03.24 02:56:48.885062 [ 2836175 ] {} <Information> NuRaftStateManager: Create ForwardingConnection for 2, 127.0.0.1:8102, thread 13, 0x10a1556d8
2023.03.24 02:56:48.885088 [ 2836175 ] {} <Information> NuRaftStateManager: Create ForwardingConnection for 2, 127.0.0.1:8102, thread 14, 0x10a155898
2023.03.24 02:56:48.885125 [ 2836175 ] {} <Information> NuRaftStateManager: Create ForwardingConnection for 2, 127.0.0.1:8102, thread 15, 0x10a155a58
2023.03.24 02:56:48.885171 [ 2836175 ] {} <Information> NuRaftStateManager: Create ForwardingConnection for 3, 127.0.0.1:8102
2023.03.24 02:56:48.885201 [ 2836175 ] {} <Information> NuRaftStateManager: Create ForwardingConnection for 3, 127.0.0.1:8102, thread 0, 0x10a155c18
2023.03.24 02:56:48.885231 [ 2836175 ] {} <Information> NuRaftStateManager: Create ForwardingConnection for 3, 127.0.0.1:8102, thread 1, 0x10a155dd8
2023.03.24 02:56:48.885261 [ 2836175 ] {} <Information> NuRaftStateManager: Create ForwardingConnection for 3, 127.0.0.1:8102, thread 2, 0x10a155f98
2023.03.24 02:56:48.885290 [ 2836175 ] {} <Information> NuRaftStateManager: Create ForwardingConnection for 3, 127.0.0.1:8102, thread 3, 0x10a156158
2023.03.24 02:56:48.885461 [ 2836175 ] {} <Information> NuRaftStateManager: Create ForwardingConnection for 3, 127.0.0.1:8102, thread 4, 0x10a156318
2023.03.24 02:56:48.885499 [ 2836175 ] {} <Information> NuRaftStateManager: Create ForwardingConnection for 3, 127.0.0.1:8102, thread 5, 0x10a1564d8
2023.03.24 02:56:48.885519 [ 2836175 ] {} <Information> NuRaftStateManager: Create ForwardingConnection for 3, 127.0.0.1:8102, thread 6, 0x10a156698
2023.03.24 02:56:48.885538 [ 2836175 ] {} <Information> NuRaftStateManager: Create ForwardingConnection for 3, 127.0.0.1:8102, thread 7, 0x10a156858
2023.03.24 02:56:48.885557 [ 2836175 ] {} <Information> NuRaftStateManager: Create ForwardingConnection for 3, 127.0.0.1:8102, thread 8, 0x10a156a18
2023.03.24 02:56:48.885646 [ 2836175 ] {} <Information> NuRaftStateManager: Create ForwardingConnection for 3, 127.0.0.1:8102, thread 9, 0x10a156bd8
2023.03.24 02:56:48.885684 [ 2836175 ] {} <Information> NuRaftStateManager: Create ForwardingConnection for 3, 127.0.0.1:8102, thread 10, 0x10a156d98
2023.03.24 02:56:48.885714 [ 2836175 ] {} <Information> NuRaftStateManager: Create ForwardingConnection for 3, 127.0.0.1:8102, thread 11, 0x10a156f58
2023.03.24 02:56:48.885735 [ 2836175 ] {} <Information> NuRaftStateManager: Create ForwardingConnection for 3, 127.0.0.1:8102, thread 12, 0x10a157118
2023.03.24 02:56:48.885824 [ 2836175 ] {} <Information> NuRaftStateManager: Create ForwardingConnection for 3, 127.0.0.1:8102, thread 13, 0x10a1572d8
2023.03.24 02:56:48.885868 [ 2836175 ] {} <Information> NuRaftStateManager: Create ForwardingConnection for 3, 127.0.0.1:8102, thread 14, 0x10a157498
2023.03.24 02:56:48.885898 [ 2836175 ] {} <Information> NuRaftStateManager: Create ForwardingConnection for 3, 127.0.0.1:8102, thread 15, 0x10a157658
2023.03.24 02:56:48.885923 [ 2836175 ] {} <Information> NuRaftStateManager: raft cluster config :  127.0.0.1:8103 127.0.0.1:8103 127.0.0.1:8103
2023.03.24 02:56:48.885971 [ 2836175 ] {} <Information> FourLetterCommandFactory: Register four letter command ruok
2023.03.24 02:56:48.886006 [ 2836175 ] {} <Information> FourLetterCommandFactory: Register four letter command mntr
2023.03.24 02:56:48.886036 [ 2836175 ] {} <Information> FourLetterCommandFactory: Register four letter command conf
2023.03.24 02:56:48.886101 [ 2836175 ] {} <Information> FourLetterCommandFactory: Register four letter command cons
2023.03.24 02:56:48.886132 [ 2836175 ] {} <Information> FourLetterCommandFactory: Register four letter command wchs
2023.03.24 02:56:48.886171 [ 2836175 ] {} <Information> FourLetterCommandFactory: Register four letter command dirs
2023.03.24 02:56:48.886201 [ 2836175 ] {} <Information> FourLetterCommandFactory: Register four letter command dump
2023.03.24 02:56:48.886230 [ 2836175 ] {} <Information> FourLetterCommandFactory: Register four letter command envi
2023.03.24 02:56:48.886257 [ 2836175 ] {} <Information> FourLetterCommandFactory: Register four letter command isro
2023.03.24 02:56:48.886284 [ 2836175 ] {} <Information> FourLetterCommandFactory: Register four letter command crst
2023.03.24 02:56:48.886372 [ 2836175 ] {} <Information> FourLetterCommandFactory: Register four letter command srvr
2023.03.24 02:56:48.886405 [ 2836175 ] {} <Information> FourLetterCommandFactory: Register four letter command stat
2023.03.24 02:56:48.886431 [ 2836175 ] {} <Information> FourLetterCommandFactory: Register four letter command srst
2023.03.24 02:56:48.886455 [ 2836175 ] {} <Information> FourLetterCommandFactory: Register four letter command wchp
2023.03.24 02:56:48.886473 [ 2836175 ] {} <Information> FourLetterCommandFactory: Register four letter command wchc
2023.03.24 02:56:48.886492 [ 2836175 ] {} <Information> FourLetterCommandFactory: Register four letter command csnp
2023.03.24 02:56:48.886510 [ 2836175 ] {} <Information> FourLetterCommandFactory: Register four letter command lgif
2023.03.24 02:56:48.886528 [ 2836175 ] {} <Information> FourLetterCommandFactory: Register four letter command rqld
2023.03.24 02:56:48.887211 [ 2836175 ] {} <Information> Application: Listening for user connections on 0.0.0.0:8101
2023.03.24 02:56:48.887800 [ 2836175 ] {} <Information> Application: Listening for forwarding connections on 0.0.0.0:8102
2023.03.24 02:56:48.892659 [ 2836175 ] {} <Information> NuRaftStateManager: Create ForwardingConnection for 2, 127.0.0.1:8102
2023.03.24 02:56:48.892759 [ 2836175 ] {} <Information> NuRaftStateManager: Create ForwardingConnection for 2, 127.0.0.1:8102, thread 0, 0x10a154018
2023.03.24 02:56:48.892811 [ 2836175 ] {} <Information> NuRaftStateManager: Create ForwardingConnection for 2, 127.0.0.1:8102, thread 1, 0x10a1541d8
2023.03.24 02:56:48.892830 [ 2836175 ] {} <Information> NuRaftStateManager: Create ForwardingConnection for 2, 127.0.0.1:8102, thread 2, 0x10a154398
2023.03.24 02:56:48.892847 [ 2836175 ] {} <Information> NuRaftStateManager: Create ForwardingConnection for 2, 127.0.0.1:8102, thread 3, 0x10a154558
2023.03.24 02:56:48.892863 [ 2836175 ] {} <Information> NuRaftStateManager: Create ForwardingConnection for 2, 127.0.0.1:8102, thread 4, 0x10a154718
2023.03.24 02:56:48.892944 [ 2836175 ] {} <Information> NuRaftStateManager: Create ForwardingConnection for 2, 127.0.0.1:8102, thread 5, 0x10a1548d8
2023.03.24 02:56:48.892985 [ 2836175 ] {} <Information> NuRaftStateManager: Create ForwardingConnection for 2, 127.0.0.1:8102, thread 6, 0x10a154a98
2023.03.24 02:56:48.893008 [ 2836175 ] {} <Information> NuRaftStateManager: Create ForwardingConnection for 2, 127.0.0.1:8102, thread 7, 0x10a154c58
2023.03.24 02:56:48.893026 [ 2836175 ] {} <Information> NuRaftStateManager: Create ForwardingConnection for 2, 127.0.0.1:8102, thread 8, 0x10a154e18
2023.03.24 02:56:48.893043 [ 2836175 ] {} <Information> NuRaftStateManager: Create ForwardingConnection for 2, 127.0.0.1:8102, thread 9, 0x10a154fd8
2023.03.24 02:56:48.893350 [ 2836175 ] {} <Information> NuRaftStateManager: Create ForwardingConnection for 2, 127.0.0.1:8102, thread 10, 0x10a155198
2023.03.24 02:56:48.893389 [ 2836175 ] {} <Information> NuRaftStateManager: Create ForwardingConnection for 2, 127.0.0.1:8102, thread 11, 0x10a155358
2023.03.24 02:56:48.893407 [ 2836175 ] {} <Information> NuRaftStateManager: Create ForwardingConnection for 2, 127.0.0.1:8102, thread 12, 0x10a155518
2023.03.24 02:56:48.893425 [ 2836175 ] {} <Information> NuRaftStateManager: Create ForwardingConnection for 2, 127.0.0.1:8102, thread 13, 0x10a1556d8
2023.03.24 02:56:48.893478 [ 2836175 ] {} <Information> NuRaftStateManager: Create ForwardingConnection for 2, 127.0.0.1:8102, thread 14, 0x10a155898
2023.03.24 02:56:48.893495 [ 2836175 ] {} <Information> NuRaftStateManager: Create ForwardingConnection for 2, 127.0.0.1:8102, thread 15, 0x10a155a58
2023.03.24 02:56:48.893526 [ 2836175 ] {} <Information> NuRaftStateManager: Create ForwardingConnection for 3, 127.0.0.1:8102
2023.03.24 02:56:48.893545 [ 2836175 ] {} <Information> NuRaftStateManager: Create ForwardingConnection for 3, 127.0.0.1:8102, thread 0, 0x10a155c18
2023.03.24 02:56:48.893561 [ 2836175 ] {} <Information> NuRaftStateManager: Create ForwardingConnection for 3, 127.0.0.1:8102, thread 1, 0x10a155dd8
2023.03.24 02:56:48.893577 [ 2836175 ] {} <Information> NuRaftStateManager: Create ForwardingConnection for 3, 127.0.0.1:8102, thread 2, 0x10a155f98
2023.03.24 02:56:48.893592 [ 2836175 ] {} <Information> NuRaftStateManager: Create ForwardingConnection for 3, 127.0.0.1:8102, thread 3, 0x10a156158
2023.03.24 02:56:48.893609 [ 2836175 ] {} <Information> NuRaftStateManager: Create ForwardingConnection for 3, 127.0.0.1:8102, thread 4, 0x10a156318
2023.03.24 02:56:48.893905 [ 2836175 ] {} <Information> NuRaftStateManager: Create ForwardingConnection for 3, 127.0.0.1:8102, thread 5, 0x10a1564d8
2023.03.24 02:56:48.893958 [ 2836175 ] {} <Information> NuRaftStateManager: Create ForwardingConnection for 3, 127.0.0.1:8102, thread 6, 0x10a156698
2023.03.24 02:56:48.893990 [ 2836175 ] {} <Information> NuRaftStateManager: Create ForwardingConnection for 3, 127.0.0.1:8102, thread 7, 0x10a156858
2023.03.24 02:56:48.894015 [ 2836175 ] {} <Information> NuRaftStateManager: Create ForwardingConnection for 3, 127.0.0.1:8102, thread 8, 0x10a156a18
2023.03.24 02:56:48.894045 [ 2836175 ] {} <Information> NuRaftStateManager: Create ForwardingConnection for 3, 127.0.0.1:8102, thread 9, 0x10a156bd8
2023.03.24 02:56:48.894180 [ 2836175 ] {} <Information> NuRaftStateManager: Create ForwardingConnection for 3, 127.0.0.1:8102, thread 10, 0x10a156d98
2023.03.24 02:56:48.894234 [ 2836175 ] {} <Information> NuRaftStateManager: Create ForwardingConnection for 3, 127.0.0.1:8102, thread 11, 0x10a156f58
2023.03.24 02:56:48.894264 [ 2836175 ] {} <Information> NuRaftStateManager: Create ForwardingConnection for 3, 127.0.0.1:8102, thread 12, 0x10a157118
2023.03.24 02:56:48.894284 [ 2836175 ] {} <Information> NuRaftStateManager: Create ForwardingConnection for 3, 127.0.0.1:8102, thread 13, 0x10a1572d8
2023.03.24 02:56:48.894475 [ 2836175 ] {} <Information> NuRaftStateManager: Create ForwardingConnection for 3, 127.0.0.1:8102, thread 14, 0x10a157498
2023.03.24 02:56:48.894520 [ 2836175 ] {} <Information> NuRaftStateManager: Create ForwardingConnection for 3, 127.0.0.1:8102, thread 15, 0x10a157658
2023.03.24 02:56:48.894542 [ 2836175 ] {} <Information> NuRaftStateManager: raft cluster config :  127.0.0.1:8103 127.0.0.1:8103 127.0.0.1:8103
2023.03.24 02:56:48.895306 [ 2836175 ] {} <Information> Application: RaftKeeper started!
2023.03.24 02:56:49.380312 [ 2836205 ] {} <Error> NuRaft: 2 nodes (out of 3, 3 including learners) are not responding longer than 10000 ms, at least 2 nodes (including leader) should be alive to proceed commit
2023.03.24 02:56:49.380554 [ 2836205 ] {} <Error> NuRaft: will yield the leadership of this node
2023.03.24 02:56:49.380652 [ 2836205 ] {} <Information> NuRaft: got immediate re-elect request, resign now
2023.03.24 02:56:49.380711 [ 2836205 ] {} <Information> NuRaft: [BECOME FOLLOWER] term 1
2023.03.24 02:57:02.777243 [ 2836209 ] {} <Warning> NuRaft: Election timeout, initiate leader election
2023.03.24 02:57:02.777575 [ 2836209 ] {} <Information> NuRaft: [PRIORITY] decay, target 1 -> 1, mine 1
2023.03.24 02:57:02.777650 [ 2836209 ] {} <Information> NuRaft: [ELECTION TIMEOUT] current role: follower, log last term 1, state term 1, target p 1, my p 1, hb dead, pre-vote done
2023.03.24 02:57:02.777751 [ 2836209 ] {} <Information> NuRaft: pre-vote term (0) is different, reset it to 1
2023.03.24 02:57:02.777945 [ 2836209 ] {} <Information> NuRaft: reset RPC client for peer 3
2023.03.24 02:57:02.778021 [ 2836209 ] {} <Information> NuRaft: reset RPC client for peer 2
2023.03.24 02:57:02.778086 [ 2836209 ] {} <Information> NuRaft: [PRE-VOTE INIT] my id 1, my role candidate, term 1, log idx 1, log term 1, priority (target 1 / mine 1)
2023.03.24 02:57:02.779086 [ 2836214 ] {} <Information> NuRaft: 0x10baf1198 connected to 127.0.0.1:8103 (as a client)
2023.03.24 02:57:02.779289 [ 2836216 ] {} <Information> NuRaft: receive a incoming rpc connection
2023.03.24 02:57:02.779572 [ 2836215 ] {} <Information> NuRaft: 0x10baf1018 connected to 127.0.0.1:8103 (as a client)
2023.03.24 02:57:02.779808 [ 2836216 ] {} <Information> NuRaft: session 3 got connection from 127.0.0.1:56231 (as a server)
2023.03.24 02:57:02.780257 [ 2836216 ] {} <Information> NuRaft: receive a incoming rpc connection
2023.03.24 02:57:02.780443 [ 2836216 ] {} <Information> NuRaft: session 4 got connection from 127.0.0.1:56232 (as a server)
2023.03.24 02:57:02.780538 [ 2836221 ] {} <Information> NuRaft: [PRE-VOTE REQ] my role candidate, from peer 1, log term: req 1 / mine 1
last idx: req 1 / mine 1, term: req 1 / mine 1
HB dead
2023.03.24 02:57:02.780838 [ 2836221 ] {} <Information> NuRaft: pre-vote decision: O (grant)
2023.03.24 02:57:02.781200 [ 2836227 ] {} <Information> NuRaft: [PRE-VOTE REQ] my role candidate, from peer 1, log term: req 1 / mine 1
last idx: req 1 / mine 1, term: req 1 / mine 1
HB dead
2023.03.24 02:57:02.781514 [ 2836227 ] {} <Information> NuRaft: pre-vote decision: O (grant)
2023.03.24 02:57:02.781712 [ 2836214 ] {} <Information> NuRaft: [PRE-VOTE RESP] peer 1 (O), term 1, resp term 1, my role candidate, dead 2, live 0, abandoned 0, num voting members 3, quorum 2
2023.03.24 02:57:02.781841 [ 2836214 ] {} <Information> NuRaft: [PRE-VOTE DONE] SUCCESS, term 1
2023.03.24 02:57:02.781892 [ 2836214 ] {} <Information> NuRaft: [PRE-VOTE DONE] initiate actual vote
2023.03.24 02:57:02.782531 [ 2836214 ] {} <Information> NuRaftStateManager: save srv_state with term 2 and vote_for 1
2023.03.24 02:57:02.782598 [ 2836214 ] {} <Information> NuRaft: [VOTE INIT] my id 1, my role candidate, term 2, log idx 1, log term 1, priority (target 1 / mine 1)
2023.03.24 02:57:02.782855 [ 2836230 ] {} <Information> NuRaft: [PRE-VOTE RESP] peer 1 (O), term 1, resp term 1, my role candidate, dead 3, live 0, abandoned 0, num voting members 3, quorum 2
2023.03.24 02:57:02.783007 [ 2836230 ] {} <Information> NuRaft: [PRE-VOTE DONE] SUCCESS, term 1
2023.03.24 02:57:02.783056 [ 2836230 ] {} <Information> NuRaft: [PRE-VOTE DONE] actual vote is already initiated, do nothing
2023.03.24 02:57:02.783182 [ 2836206 ] {} <Information> NuRaft: [VOTE REQ] my role candidate, from peer 1, log term: req 1 / mine 1
last idx: req 1 / mine 1, term: req 2 / mine 2
priority: target 1 / mine 1, voted_for 1
2023.03.24 02:57:02.783352 [ 2836206 ] {} <Information> NuRaft: decision: O (grant), voted_for 1, term 2
2023.03.24 02:57:02.784033 [ 2836206 ] {} <Information> NuRaftStateManager: save srv_state with term 2 and vote_for 1
2023.03.24 02:57:02.784347 [ 2836234 ] {} <Information> NuRaft: [VOTE REQ] my role candidate, from peer 1, log term: req 1 / mine 1
last idx: req 1 / mine 1, term: req 2 / mine 2
priority: target 1 / mine 1, voted_for 1
2023.03.24 02:57:02.784493 [ 2836234 ] {} <Information> NuRaft: decision: O (grant), voted_for 1, term 2
2023.03.24 02:57:02.785071 [ 2836234 ] {} <Information> NuRaftStateManager: save srv_state with term 2 and vote_for 1
2023.03.24 02:57:02.785358 [ 2836207 ] {} <Information> NuRaft: [VOTE RESP] peer 1 (O), resp term 2, my role candidate, granted 2, responded 2, num voting members 3, quorum 2
2023.03.24 02:57:02.785504 [ 2836207 ] {} <Information> NuRaft: Server is elected as leader for term 2
2023.03.24 02:57:02.785567 [ 2836207 ] {} <Information> NuRaft: number of pending commit elements: 0
2023.03.24 02:57:02.785609 [ 2836207 ] {} <Information> NuRaft: state machine commit index 0, precommit index 1, last log index 1
2023.03.24 02:57:02.785763 [ 2836207 ] {} <Information> NuRaft: found uncommitted config at 1, size 112
2023.03.24 02:57:02.785826 [ 2836207 ] {} <Information> NuRaft: [BECOME LEADER] appended new config at 2
2023.03.24 02:57:02.786372 [ 2836207 ] {} <Information> NuRaft:   === LEADER (term 2) ===
2023.03.24 02:57:02.786486 [ 2836207 ] {} <Warning> NuRaft: Receive AppendEntriesRequest from another leader (1) with same term, there must be a bug. Ignore it instead of exit.
2023.03.24 02:57:02.786585 [ 2836207 ] {} <Warning> NuRaft: no response is returned from raft message handler
2023.03.24 02:57:02.786668 [ 2836213 ] {} <Warning> NuRaft: Receive AppendEntriesRequest from another leader (1) with same term, there must be a bug. Ignore it instead of exit.
2023.03.24 02:57:02.786824 [ 2836229 ] {} <Information> NuRaft: Election completed, will ignore the voting result from this server
2023.03.24 02:57:02.786969 [ 2836213 ] {} <Warning> NuRaft: no response is returned from raft message handler
2023.03.24 02:57:02.787115 [ 2836217 ] {} <Warning> NuRaft: peer (3) response error: failed to read response to peer 3, 127.0.0.1:8103, error 2, End of file
2023.03.24 02:57:02.787410 [ 2836222 ] {} <Warning> NuRaft: peer (2) response error: failed to read response to peer 2, 127.0.0.1:8103, error 2, End of file
2023.03.24 02:57:03.287074 [ 2836219 ] {} <Error> NuRaft: 2 nodes (out of 3, 3 including learners) are not responding longer than 10000 ms, at least 2 nodes (including leader) should be alive to proceed commit
2023.03.24 02:57:03.287236 [ 2836219 ] {} <Error> NuRaft: will yield the leadership of this node
2023.03.24 02:57:03.287287 [ 2836219 ] {} <Information> NuRaft: got immediate re-elect request, resign now
2023.03.24 02:57:03.287318 [ 2836219 ] {} <Information> NuRaft: [BECOME FOLLOWER] term 2
2023.03.24 02:57:18.602044 [ 2836225 ] {} <Warning> NuRaft: Election timeout, initiate leader election
2023.03.24 02:57:18.602360 [ 2836225 ] {} <Information> NuRaft: [PRIORITY] decay, target 1 -> 1, mine 1
2023.03.24 02:57:18.602388 [ 2836225 ] {} <Information> NuRaft: [ELECTION TIMEOUT] current role: follower, log last term 2, state term 2, target p 1, my p 1, hb dead, pre-vote done
2023.03.24 02:57:18.602439 [ 2836225 ] {} <Information> NuRaft: pre-vote term (1) is different, reset it to 2
2023.03.24 02:57:18.602467 [ 2836225 ] {} <Information> NuRaft: reset RPC client for peer 3
2023.03.24 02:57:18.602491 [ 2836225 ] {} <Information> NuRaft: reset RPC client for peer 2
2023.03.24 02:57:18.602514 [ 2836225 ] {} <Information> NuRaft: [PRE-VOTE INIT] my id 1, my role candidate, term 2, log idx 2, log term 2, priority (target 1 / mine 1)
2023.03.24 02:57:18.603561 [ 2836221 ] {} <Information> NuRaft: 0x10d2a2018 connected to 127.0.0.1:8103 (as a client)
2023.03.24 02:57:18.603720 [ 2836214 ] {} <Information> NuRaft: receive a incoming rpc connection
2023.03.24 02:57:18.603833 [ 2836228 ] {} <Information> NuRaft: 0x10d2a2198 connected to 127.0.0.1:8103 (as a client)
2023.03.24 02:57:18.604073 [ 2836214 ] {} <Information> NuRaft: session 5 got connection from 127.0.0.1:56269 (as a server)
2023.03.24 02:57:18.604259 [ 2836226 ] {} <Information> NuRaft: [PRE-VOTE REQ] my role candidate, from peer 1, log term: req 2 / mine 2
last idx: req 2 / mine 2, term: req 2 / mine 2
HB dead
2023.03.24 02:57:18.604393 [ 2836226 ] {} <Information> NuRaft: pre-vote decision: O (grant)
2023.03.24 02:57:18.604460 [ 2836214 ] {} <Information> NuRaft: receive a incoming rpc connection
2023.03.24 02:57:18.604554 [ 2836214 ] {} <Information> NuRaft: session 6 got connection from 127.0.0.1:56270 (as a server)
2023.03.24 02:57:18.604663 [ 2836233 ] {} <Information> NuRaft: [PRE-VOTE RESP] peer 1 (O), term 2, resp term 2, my role candidate, dead 2, live 0, abandoned 0, num voting members 3, quorum 2
2023.03.24 02:57:18.604754 [ 2836233 ] {} <Information> NuRaft: [PRE-VOTE DONE] SUCCESS, term 2
2023.03.24 02:57:18.604784 [ 2836233 ] {} <Information> NuRaft: [PRE-VOTE DONE] initiate actual vote
2023.03.24 02:57:18.605431 [ 2836233 ] {} <Information> NuRaftStateManager: save srv_state with term 3 and vote_for 1
2023.03.24 02:57:18.605560 [ 2836233 ] {} <Information> NuRaft: [VOTE INIT] my id 1, my role candidate, term 3, log idx 2, log term 2, priority (target 1 / mine 1)
2023.03.24 02:57:18.605705 [ 2836233 ] {} <Warning> NuRaft: failed to send vote request: peer 2 (127.0.0.1:8103) is busy
2023.03.24 02:57:18.605869 [ 2836214 ] {} <Information> NuRaft: [PRE-VOTE REQ] my role candidate, from peer 1, log term: req 2 / mine 2
last idx: req 2 / mine 2, term: req 2 / mine 3
HB dead
2023.03.24 02:57:18.605950 [ 2836214 ] {} <Information> NuRaft: pre-vote decision: O (grant)
2023.03.24 02:57:18.606024 [ 2836206 ] {} <Information> NuRaft: [VOTE REQ] my role candidate, from peer 1, log term: req 2 / mine 2
last idx: req 2 / mine 2, term: req 3 / mine 3
priority: target 1 / mine 1, voted_for 1
2023.03.24 02:57:18.606085 [ 2836206 ] {} <Information> NuRaft: decision: O (grant), voted_for 1, term 3
2023.03.24 02:57:18.606459 [ 2836206 ] {} <Information> NuRaftStateManager: save srv_state with term 3 and vote_for 1
2023.03.24 02:57:18.606540 [ 2836205 ] {} <Information> NuRaft: [PRE-VOTE RESP] peer 1 (O), term 2, resp term 2, my role candidate, dead 3, live 0, abandoned 0, num voting members 3, quorum 2
2023.03.24 02:57:18.606600 [ 2836205 ] {} <Information> NuRaft: [PRE-VOTE DONE] SUCCESS, term 2
2023.03.24 02:57:18.606622 [ 2836205 ] {} <Information> NuRaft: [PRE-VOTE DONE] actual vote is already initiated, do nothing
2023.03.24 02:57:18.606759 [ 2836211 ] {} <Information> NuRaft: [VOTE RESP] peer 1 (O), resp term 3, my role candidate, granted 2, responded 2, num voting members 3, quorum 2
2023.03.24 02:57:18.606809 [ 2836211 ] {} <Information> NuRaft: Server is elected as leader for term 3
2023.03.24 02:57:18.606836 [ 2836211 ] {} <Information> NuRaft: number of pending commit elements: 0
2023.03.24 02:57:18.606857 [ 2836211 ] {} <Information> NuRaft: state machine commit index 0, precommit index 2, last log index 2
2023.03.24 02:57:18.606889 [ 2836211 ] {} <Information> NuRaft: found uncommitted config at 1, size 112
2023.03.24 02:57:18.606945 [ 2836211 ] {} <Information> NuRaft: found uncommitted config at 2, size 112
2023.03.24 02:57:18.607033 [ 2836211 ] {} <Information> NuRaft: [BECOME LEADER] appended new config at 3
2023.03.24 02:57:18.607445 [ 2836211 ] {} <Information> NuRaft:   === LEADER (term 3) ===
2023.03.24 02:57:18.607616 [ 2836207 ] {} <Warning> NuRaft: Receive AppendEntriesRequest from another leader (1) with same term, there must be a bug. Ignore it instead of exit.
2023.03.24 02:57:18.607679 [ 2836207 ] {} <Warning> NuRaft: no response is returned from raft message handler
2023.03.24 02:57:18.607728 [ 2836224 ] {} <Warning> NuRaft: Receive AppendEntriesRequest from another leader (1) with same term, there must be a bug. Ignore it instead of exit.
2023.03.24 02:57:18.607863 [ 2836224 ] {} <Warning> NuRaft: no response is returned from raft message handler
2023.03.24 02:57:18.608018 [ 2836213 ] {} <Warning> NuRaft: peer (2) response error: failed to read response to peer 2, 127.0.0.1:8103, error 2, End of file
2023.03.24 02:57:18.608200 [ 2836215 ] {} <Warning> NuRaft: peer (3) response error: failed to read response to peer 3, 127.0.0.1:8103, error 2, End of file
2023.03.24 02:57:19.108176 [ 2836225 ] {} <Error> NuRaft: 2 nodes (out of 3, 3 including learners) are not responding longer than 10000 ms, at least 2 nodes (including leader) should be alive to proceed commit
2023.03.24 02:57:19.108486 [ 2836225 ] {} <Error> NuRaft: will yield the leadership of this node
2023.03.24 02:57:19.108568 [ 2836225 ] {} <Information> NuRaft: got immediate re-elect request, resign now
2023.03.24 02:57:19.108620 [ 2836225 ] {} <Information> NuRaft: [BECOME FOLLOWER] term 3
2023.03.24 02:57:34.068779 [ 2836227 ] {} <Warning> NuRaft: Election timeout, initiate leader election
2023.03.24 02:57:34.069064 [ 2836227 ] {} <Information> NuRaft: [PRIORITY] decay, target 1 -> 1, mine 1
2023.03.24 02:57:34.069137 [ 2836227 ] {} <Information> NuRaft: [ELECTION TIMEOUT] current role: follower, log last term 3, state term 3, target p 1, my p 1, hb dead, pre-vote done
2023.03.24 02:57:34.069300 [ 2836227 ] {} <Information> NuRaft: pre-vote term (2) is different, reset it to 3
2023.03.24 02:57:34.069357 [ 2836227 ] {} <Information> NuRaft: reset RPC client for peer 3
2023.03.24 02:57:34.069411 [ 2836227 ] {} <Information> NuRaft: reset RPC client for peer 2
2023.03.24 02:57:34.069465 [ 2836227 ] {} <Information> NuRaft: [PRE-VOTE INIT] my id 1, my role candidate, term 3, log idx 3, log term 3, priority (target 1 / mine 1)
2023.03.24 02:57:34.070591 [ 2836220 ] {} <Information> NuRaft: 0x10f209198 connected to 127.0.0.1:8103 (as a client)
2023.03.24 02:57:34.070789 [ 2836228 ] {} <Information> NuRaft: 0x10f209018 connected to 127.0.0.1:8103 (as a client)
2023.03.24 02:57:34.070943 [ 2836230 ] {} <Information> NuRaft: receive a incoming rpc connection
2023.03.24 02:57:34.071057 [ 2836230 ] {} <Information> NuRaft: session 7 got connection from 127.0.0.1:56309 (as a server)
2023.03.24 02:57:34.071271 [ 2836234 ] {} <Information> NuRaft: [PRE-VOTE REQ] my role candidate, from peer 1, log term: req 3 / mine 3
last idx: req 3 / mine 3, term: req 3 / mine 3
HB dead
2023.03.24 02:57:34.071452 [ 2836234 ] {} <Information> NuRaft: pre-vote decision: O (grant)
2023.03.24 02:57:34.071574 [ 2836230 ] {} <Information> NuRaft: receive a incoming rpc connection
2023.03.24 02:57:34.071738 [ 2836230 ] {} <Information> NuRaft: session 8 got connection from 127.0.0.1:56310 (as a server)
2023.03.24 02:57:34.071829 [ 2836226 ] {} <Information> NuRaft: [PRE-VOTE RESP] peer 1 (O), term 3, resp term 3, my role candidate, dead 2, live 0, abandoned 0, num voting members 3, quorum 2
2023.03.24 02:57:34.071968 [ 2836226 ] {} <Information> NuRaft: [PRE-VOTE DONE] SUCCESS, term 3
2023.03.24 02:57:34.072015 [ 2836226 ] {} <Information> NuRaft: [PRE-VOTE DONE] initiate actual vote
2023.03.24 02:57:34.073311 [ 2836226 ] {} <Information> NuRaftStateManager: save srv_state with term 4 and vote_for 1
2023.03.24 02:57:34.073424 [ 2836226 ] {} <Information> NuRaft: [VOTE INIT] my id 1, my role candidate, term 4, log idx 3, log term 3, priority (target 1 / mine 1)
2023.03.24 02:57:34.073463 [ 2836226 ] {} <Warning> NuRaft: failed to send vote request: peer 3 (127.0.0.1:8103) is busy
2023.03.24 02:57:34.073671 [ 2836209 ] {} <Information> NuRaft: [PRE-VOTE REQ] my role candidate, from peer 1, log term: req 3 / mine 3
last idx: req 3 / mine 3, term: req 3 / mine 4
HB dead
2023.03.24 02:57:34.073801 [ 2836209 ] {} <Information> NuRaft: pre-vote decision: O (grant)
2023.03.24 02:57:34.073890 [ 2836214 ] {} <Information> NuRaft: [VOTE REQ] my role candidate, from peer 1, log term: req 3 / mine 3
last idx: req 3 / mine 3, term: req 4 / mine 4
priority: target 1 / mine 1, voted_for 1
2023.03.24 02:57:34.073970 [ 2836214 ] {} <Information> NuRaft: decision: O (grant), voted_for 1, term 4
2023.03.24 02:57:34.074643 [ 2836214 ] {} <Information> NuRaftStateManager: save srv_state with term 4 and vote_for 1
2023.03.24 02:57:34.074800 [ 2836206 ] {} <Information> NuRaft: [PRE-VOTE RESP] peer 1 (O), term 3, resp term 3, my role candidate, dead 3, live 0, abandoned 0, num voting members 3, quorum 2
2023.03.24 02:57:34.074908 [ 2836206 ] {} <Information> NuRaft: [PRE-VOTE DONE] SUCCESS, term 3
2023.03.24 02:57:34.074956 [ 2836206 ] {} <Information> NuRaft: [PRE-VOTE DONE] actual vote is already initiated, do nothing
2023.03.24 02:57:34.075027 [ 2836229 ] {} <Information> NuRaft: [VOTE RESP] peer 1 (O), resp term 4, my role candidate, granted 2, responded 2, num voting members 3, quorum 2
2023.03.24 02:57:34.075118 [ 2836229 ] {} <Information> NuRaft: Server is elected as leader for term 4
2023.03.24 02:57:34.075182 [ 2836229 ] {} <Information> NuRaft: number of pending commit elements: 0
2023.03.24 02:57:34.075268 [ 2836229 ] {} <Information> NuRaft: state machine commit index 0, precommit index 3, last log index 3
2023.03.24 02:57:34.075344 [ 2836229 ] {} <Information> NuRaft: found uncommitted config at 1, size 112
2023.03.24 02:57:34.075395 [ 2836229 ] {} <Information> NuRaft: found uncommitted config at 2, size 112
2023.03.24 02:57:34.075498 [ 2836229 ] {} <Information> NuRaft: found uncommitted config at 3, size 112
2023.03.24 02:57:34.075545 [ 2836229 ] {} <Information> NuRaft: [BECOME LEADER] appended new config at 4
2023.03.24 02:57:34.076051 [ 2836229 ] {} <Information> NuRaft:   === LEADER (term 4) ===
2023.03.24 02:57:34.076222 [ 2836222 ] {} <Warning> NuRaft: Receive AppendEntriesRequest from another leader (1) with same term, there must be a bug. Ignore it instead of exit.
2023.03.24 02:57:34.076338 [ 2836222 ] {} <Warning> NuRaft: no response is returned from raft message handler
2023.03.24 02:57:34.076400 [ 2836215 ] {} <Warning> NuRaft: Receive AppendEntriesRequest from another leader (1) with same term, there must be a bug. Ignore it instead of exit.
2023.03.24 02:57:34.076535 [ 2836215 ] {} <Warning> NuRaft: no response is returned from raft message handler
2023.03.24 02:57:34.076692 [ 2836213 ] {} <Warning> NuRaft: peer (2) response error: failed to read response to peer 2, 127.0.0.1:8103, error 2, End of file
2023.03.24 02:57:34.076884 [ 2836227 ] {} <Warning> NuRaft: peer (3) response error: failed to read response to peer 3, 127.0.0.1:8103, error 2, End of file
2023.03.24 02:57:34.576686 [ 2836232 ] {} <Error> NuRaft: 2 nodes (out of 3, 3 including learners) are not responding longer than 10000 ms, at least 2 nodes (including leader) should be alive to proceed commit
2023.03.24 02:57:34.577036 [ 2836232 ] {} <Error> NuRaft: will yield the leadership of this node
2023.03.24 02:57:34.577131 [ 2836232 ] {} <Information> NuRaft: got immediate re-elect request, resign now
2023.03.24 02:57:34.577189 [ 2836232 ] {} <Information> NuRaft: [BECOME FOLLOWER] term 4
2023.03.24 02:57:52.469107 [ 2836221 ] {} <Warning> NuRaft: Election timeout, initiate leader election
2023.03.24 02:57:52.469208 [ 2836221 ] {} <Information> NuRaft: [PRIORITY] decay, target 1 -> 1, mine 1
2023.03.24 02:57:52.469235 [ 2836221 ] {} <Information> NuRaft: [ELECTION TIMEOUT] current role: follower, log last term 4, state term 4, target p 1, my p 1, hb dead, pre-vote done
2023.03.24 02:57:52.469278 [ 2836221 ] {} <Information> NuRaft: pre-vote term (3) is different, reset it to 4
2023.03.24 02:57:52.469300 [ 2836221 ] {} <Information> NuRaft: reset RPC client for peer 3
2023.03.24 02:57:52.469324 [ 2836221 ] {} <Information> NuRaft: reset RPC client for peer 2
2023.03.24 02:57:52.471454 [ 2836221 ] {} <Information> NuRaft: [PRE-VOTE INIT] my id 1, my role candidate, term 4, log idx 4, log term 4, priority (target 1 / mine 1)
2023.03.24 02:57:52.472534 [ 2836220 ] {} <Information> NuRaft: 0x10f1d8018 connected to 127.0.0.1:8103 (as a client)
2023.03.24 02:57:52.472651 [ 2836234 ] {} <Information> NuRaft: receive a incoming rpc connection
2023.03.24 02:57:52.472743 [ 2836218 ] {} <Information> NuRaft: 0x10f1d8198 connected to 127.0.0.1:8103 (as a client)
2023.03.24 02:57:52.472861 [ 2836234 ] {} <Information> NuRaft: session 9 got connection from 127.0.0.1:56356 (as a server)
2023.03.24 02:57:52.472972 [ 2836234 ] {} <Information> NuRaft: receive a incoming rpc connection
2023.03.24 02:57:52.473016 [ 2836234 ] {} <Information> NuRaft: session 10 got connection from 127.0.0.1:56357 (as a server)
2023.03.24 02:57:52.473103 [ 2836233 ] {} <Information> NuRaft: [PRE-VOTE REQ] my role candidate, from peer 1, log term: req 4 / mine 4
last idx: req 4 / mine 4, term: req 4 / mine 4
HB dead
2023.03.24 02:57:52.473183 [ 2836233 ] {} <Information> NuRaft: pre-vote decision: O (grant)
2023.03.24 02:57:52.473274 [ 2836234 ] {} <Information> NuRaft: [PRE-VOTE REQ] my role candidate, from peer 1, log term: req 4 / mine 4
last idx: req 4 / mine 4, term: req 4 / mine 4
HB dead
2023.03.24 02:57:52.473313 [ 2836234 ] {} <Information> NuRaft: pre-vote decision: O (grant)
2023.03.24 02:57:52.473386 [ 2836210 ] {} <Information> NuRaft: [PRE-VOTE RESP] peer 1 (O), term 4, resp term 4, my role candidate, dead 2, live 0, abandoned 0, num voting members 3, quorum 2
2023.03.24 02:57:52.473433 [ 2836210 ] {} <Information> NuRaft: [PRE-VOTE DONE] SUCCESS, term 4
2023.03.24 02:57:52.473478 [ 2836210 ] {} <Information> NuRaft: [PRE-VOTE DONE] initiate actual vote
2023.03.24 02:57:52.473952 [ 2836210 ] {} <Information> NuRaftStateManager: save srv_state with term 5 and vote_for 1
2023.03.24 02:57:52.474018 [ 2836210 ] {} <Information> NuRaft: [VOTE INIT] my id 1, my role candidate, term 5, log idx 4, log term 4, priority (target 1 / mine 1)
2023.03.24 02:57:52.474228 [ 2836212 ] {} <Information> NuRaft: [PRE-VOTE RESP] peer 1 (O), term 4, resp term 4, my role candidate, dead 3, live 0, abandoned 0, num voting members 3, quorum 2
2023.03.24 02:57:52.474323 [ 2836212 ] {} <Information> NuRaft: [PRE-VOTE DONE] SUCCESS, term 4
2023.03.24 02:57:52.474350 [ 2836212 ] {} <Information> NuRaft: [PRE-VOTE DONE] actual vote is already initiated, do nothing
2023.03.24 02:57:52.475191 [ 2836214 ] {} <Information> NuRaft: [VOTE REQ] my role candidate, from peer 1, log term: req 4 / mine 4
last idx: req 4 / mine 4, term: req 5 / mine 5
priority: target 1 / mine 1, voted_for 1
2023.03.24 02:57:52.475247 [ 2836214 ] {} <Information> NuRaft: decision: O (grant), voted_for 1, term 5
2023.03.24 02:57:52.475506 [ 2836214 ] {} <Information> NuRaftStateManager: save srv_state with term 5 and vote_for 1
2023.03.24 02:57:52.475615 [ 2836219 ] {} <Information> NuRaft: [VOTE REQ] my role candidate, from peer 1, log term: req 4 / mine 4
last idx: req 4 / mine 4, term: req 5 / mine 5
priority: target 1 / mine 1, voted_for 1
2023.03.24 02:57:52.475720 [ 2836219 ] {} <Information> NuRaft: decision: O (grant), voted_for 1, term 5
2023.03.24 02:57:52.476071 [ 2836219 ] {} <Information> NuRaftStateManager: save srv_state with term 5 and vote_for 1
2023.03.24 02:57:52.476170 [ 2836207 ] {} <Information> NuRaft: [VOTE RESP] peer 1 (O), resp term 5, my role candidate, granted 2, responded 2, num voting members 3, quorum 2
2023.03.24 02:57:52.476222 [ 2836207 ] {} <Information> NuRaft: Server is elected as leader for term 5
2023.03.24 02:57:52.476248 [ 2836207 ] {} <Information> NuRaft: number of pending commit elements: 0
2023.03.24 02:57:52.476265 [ 2836207 ] {} <Information> NuRaft: state machine commit index 0, precommit index 4, last log index 4
2023.03.24 02:57:52.476292 [ 2836207 ] {} <Information> NuRaft: found uncommitted config at 1, size 112
2023.03.24 02:57:52.476313 [ 2836207 ] {} <Information> NuRaft: found uncommitted config at 2, size 112
2023.03.24 02:57:52.476333 [ 2836207 ] {} <Information> NuRaft: found uncommitted config at 3, size 112
2023.03.24 02:57:52.476349 [ 2836207 ] {} <Information> NuRaft: found uncommitted config at 4, size 112
2023.03.24 02:57:52.476371 [ 2836207 ] {} <Information> NuRaft: [BECOME LEADER] appended new config at 5
2023.03.24 02:57:52.476837 [ 2836207 ] {} <Information> NuRaft:   === LEADER (term 5) ===
2023.03.24 02:57:52.476916 [ 2836225 ] {} <Information> NuRaft: Election completed, will ignore the voting result from this server
2023.03.24 02:57:52.477027 [ 2836221 ] {} <Warning> NuRaft: Receive AppendEntriesRequest from another leader (1) with same term, there must be a bug. Ignore it instead of exit.
2023.03.24 02:57:52.477091 [ 2836221 ] {} <Warning> NuRaft: no response is returned from raft message handler
2023.03.24 02:57:52.477162 [ 2836208 ] {} <Warning> NuRaft: Receive AppendEntriesRequest from another leader (1) with same term, there must be a bug. Ignore it instead of exit.
2023.03.24 02:57:52.477244 [ 2836208 ] {} <Warning> NuRaft: no response is returned from raft message handler
2023.03.24 02:57:52.477327 [ 2836207 ] {} <Warning> NuRaft: peer (3) response error: failed to read response to peer 3, 127.0.0.1:8103, error 2, End of file
2023.03.24 02:57:52.477533 [ 2836204 ] {} <Warning> NuRaft: peer (2) response error: failed to read response to peer 2, 127.0.0.1:8103, error 2, End of file
2023.03.24 02:57:52.977313 [ 2836230 ] {} <Error> NuRaft: 2 nodes (out of 3, 3 including learners) are not responding longer than 10000 ms, at least 2 nodes (including leader) should be alive to proceed commit
2023.03.24 02:57:52.977531 [ 2836230 ] {} <Error> NuRaft: will yield the leadership of this node
2023.03.24 02:57:52.977837 [ 2836230 ] {} <Information> NuRaft: got immediate re-elect request, resign now
2023.03.24 02:57:52.977884 [ 2836230 ] {} <Information> NuRaft: [BECOME FOLLOWER] term 5
2023.03.24 02:58:06.312654 [ 2836220 ] {} <Warning> NuRaft: Election timeout, initiate leader election
2023.03.24 02:58:06.312941 [ 2836220 ] {} <Information> NuRaft: [PRIORITY] decay, target 1 -> 1, mine 1
2023.03.24 02:58:06.313020 [ 2836220 ] {} <Information> NuRaft: [ELECTION TIMEOUT] current role: follower, log last term 5, state term 5, target p 1, my p 1, hb dead, pre-vote done
2023.03.24 02:58:06.313147 [ 2836220 ] {} <Information> NuRaft: pre-vote term (4) is different, reset it to 5
2023.03.24 02:58:06.313226 [ 2836220 ] {} <Information> NuRaft: reset RPC client for peer 3
2023.03.24 02:58:06.313320 [ 2836220 ] {} <Information> NuRaft: reset RPC client for peer 2
2023.03.24 02:58:06.313395 [ 2836220 ] {} <Information> NuRaft: [PRE-VOTE INIT] my id 1, my role candidate, term 5, log idx 5, log term 5, priority (target 1 / mine 1)
2023.03.24 02:58:06.314632 [ 2836223 ] {} <Information> NuRaft: 0x10d2e4198 connected to 127.0.0.1:8103 (as a client)
2023.03.24 02:58:06.314985 [ 2836234 ] {} <Information> NuRaft: 0x10d2e4018 connected to 127.0.0.1:8103 (as a client)
2023.03.24 02:58:06.315265 [ 2836217 ] {} <Information> NuRaft: receive a incoming rpc connection
2023.03.24 02:58:06.315467 [ 2836217 ] {} <Information> NuRaft: session 11 got connection from 127.0.0.1:56393 (as a server)
2023.03.24 02:58:06.315673 [ 2836210 ] {} <Information> NuRaft: receive a incoming rpc connection
2023.03.24 02:58:06.315983 [ 2836217 ] {} <Information> NuRaft: [PRE-VOTE REQ] my role candidate, from peer 1, log term: req 5 / mine 5
last idx: req 5 / mine 5, term: req 5 / mine 5
HB dead
2023.03.24 02:58:06.316239 [ 2836217 ] {} <Information> NuRaft: pre-vote decision: O (grant)
2023.03.24 02:58:06.316621 [ 2836210 ] {} <Information> NuRaft: session 12 got connection from 127.0.0.1:56394 (as a server)
2023.03.24 02:58:06.316810 [ 2836211 ] {} <Information> NuRaft: [PRE-VOTE RESP] peer 1 (O), term 5, resp term 5, my role candidate, dead 2, live 0, abandoned 0, num voting members 3, quorum 2
2023.03.24 02:58:06.316953 [ 2836211 ] {} <Information> NuRaft: [PRE-VOTE DONE] SUCCESS, term 5
2023.03.24 02:58:06.317018 [ 2836211 ] {} <Information> NuRaft: [PRE-VOTE DONE] initiate actual vote
2023.03.24 02:58:06.317835 [ 2836211 ] {} <Information> NuRaftStateManager: save srv_state with term 6 and vote_for 1
2023.03.24 02:58:06.317921 [ 2836211 ] {} <Information> NuRaft: [VOTE INIT] my id 1, my role candidate, term 6, log idx 5, log term 5, priority (target 1 / mine 1)
2023.03.24 02:58:06.317962 [ 2836211 ] {} <Warning> NuRaft: failed to send vote request: peer 3 (127.0.0.1:8103) is busy
2023.03.24 02:58:06.318314 [ 2836231 ] {} <Information> NuRaft: [PRE-VOTE REQ] my role candidate, from peer 1, log term: req 5 / mine 5
last idx: req 5 / mine 5, term: req 5 / mine 6
HB dead
2023.03.24 02:58:06.318465 [ 2836231 ] {} <Information> NuRaft: pre-vote decision: O (grant)
2023.03.24 02:58:06.318569 [ 2836229 ] {} <Information> NuRaft: [VOTE REQ] my role candidate, from peer 1, log term: req 5 / mine 5
last idx: req 5 / mine 5, term: req 6 / mine 6
priority: target 1 / mine 1, voted_for 1
2
...

It seems that the Raft cluster has failed to maintain heartbeats, resulting in re-elections.

Has this situation occurred before? If it is confirmed to be a reproducible situation, I am willing to participate in resolving it.

RaftKeeper server should take a session timeout lower bound.

Description
Client negotiate session timeout with server to determine a real timeout. And server should have a lower and upper bound, but not it only has a upper bound raft_settings.session_timeout_ms.

Solution
Add a lower bound like raft_settings.min_session_timeout_ms. And you could rename session_timeout_ms to max_session_timeout_ms

Code of Conduct
The Code of Conduct helps create a safe space for everyone. We require that everyone agrees to it.

[ ] I agree to follow this project's Code of Conduct *

Snapshot should keep last log term.

description

Snapshot should keep last log term

Version *
2.0.0

ThreadSanitizer: heap-use-after-free

RaftKeeper version

v2.0.2

Description

Found in test case test_restore_from_snapshot

Error occurs when a new connection arrive. Socket acceptor thread will register event handler(3 handlers) and when the first readable event handler registerd, reactor thread may receive a readable event and handle the event, if handling failed, it will delete socket, then data race occurs.

Details:

2023-10-10T14:29:33.4407384Z WARNING: ThreadSanitizer: heap-use-after-free (pid=8)
2023-10-10T14:29:33.4407875Z   Read of size 8 at 0x7b6000015968 by thread T108 (mutexes: write M0, write M1):
2023-10-10T14:29:33.4408550Z     #0 Poco::Net::Socket::impl() const /home/runner/work/RaftKeeper/RaftKeeper/contrib/poco/Net/include/Poco/Net/Socket.h:780:9 (raftkeeper+0x86725b) (BuildId: 43558d85c161f514)
2023-10-10T14:29:33.4409483Z     #1 RK::PollSetImpl::add(Poco::Net::Socket const&, int) /home/runner/work/RaftKeeper/RaftKeeper/src/Common/NIO/PollSet.cpp:61:39 (raftkeeper+0x86725b)
2023-10-10T14:29:33.4410269Z     #2 RK::PollSet::add(Poco::Net::Socket const&, int) /home/runner/work/RaftKeeper/RaftKeeper/src/Common/NIO/PollSet.cpp:455:13 (raftkeeper+0x866a11) (BuildId: 43558d85c161f514)
2023-10-10T14:29:33.4411296Z     #3 RK::SocketReactor::addEventHandlers(Poco::Net::Socket const&, std::__1::vector<Poco::AbstractObserver*, std::__1::allocator<Poco::AbstractObserver*> > const&) /home/runner/work/RaftKeeper/RaftKeeper/src/Common/NIO/SocketReactor.cpp:180:28 (raftkeeper+0x8649c3) (BuildId: 43558d85c161f514)
2023-10-10T14:29:33.4412435Z     #4 RK::ConnectionHandler::ConnectionHandler(RK::Context&, Poco::Net::StreamSocket&, RK::SocketReactor&) /home/runner/work/RaftKeeper/RaftKeeper/src/Service/ConnectionHandler.cpp:90:13 (raftkeeper+0x889386) (BuildId: 43558d85c161f514)
2023-10-10T14:29:33.4418094Z     #5 RK::SvsSocketAcceptor<RK::ConnectionHandler, RK::SocketReactor>::createServiceHandler(Poco::Net::StreamSocket&) /home/runner/work/RaftKeeper/RaftKeeper/src/Common/NIO/SvsSocketAcceptor.h:161:25 (raftkeeper+0x7e7cc5) (BuildId: 43558d85c161f514)
2023-10-10T14:29:33.4419222Z     #6 RK::SvsSocketAcceptor<RK::ConnectionHandler, RK::SocketReactor>::onAccept(RK::ReadableNotification*) /home/runner/work/RaftKeeper/RaftKeeper/src/Common/NIO/SvsSocketAcceptor.h:138:9 (raftkeeper+0x7e7484) (BuildId: 43558d85c161f514)
2023-10-10T14:29:33.4420398Z     #7 Poco::Observer<RK::SvsSocketAcceptor<RK::ConnectionHandler, RK::SocketReactor>, RK::ReadableNotification>::notify(Poco::Notification*) const /home/runner/work/RaftKeeper/RaftKeeper/contrib/poco/Foundation/include/Poco/Observer.h:84:5 (raftkeeper+0x7e850e) (BuildId: 43558d85c161f514)
2023-10-10T14:29:33.4421537Z     #8 Poco::NotificationCenter::postNotification(Poco::AutoPtr<Poco::Notification>) /home/runner/work/RaftKeeper/RaftKeeper/contrib/poco/Foundation/src/NotificationCenter.cpp:76:6 (raftkeeper+0xc6b447) (BuildId: 43558d85c161f514)
2023-10-10T14:29:33.4422498Z     #9 RK::SocketNotifier::dispatch(RK::SocketNotification*) /home/runner/work/RaftKeeper/RaftKeeper/src/Common/NIO/SocketNotifier.cpp:70:13 (raftkeeper+0x86a2ab) (BuildId: 43558d85c161f514)
2023-10-10T14:29:33.4423443Z     #10 RK::SocketReactor::dispatch(Poco::AutoPtr<RK::SocketNotifier>&, RK::SocketNotification*) /home/runner/work/RaftKeeper/RaftKeeper/src/Common/NIO/SocketReactor.cpp:295:20 (raftkeeper+0x8654d0) (BuildId: 43558d85c161f514)
2023-10-10T14:29:33.4424423Z     #11 RK::SocketReactor::dispatch(Poco::Net::Socket const&, RK::SocketNotification*) /home/runner/work/RaftKeeper/RaftKeeper/src/Common/NIO/SocketReactor.cpp:271:5 (raftkeeper+0x862fb6) (BuildId: 43558d85c161f514)
2023-10-10T14:29:33.4425230Z     #12 RK::SocketReactor::run() /home/runner/work/RaftKeeper/RaftKeeper/src/Common/NIO/SocketReactor.cpp:81:29 (raftkeeper+0x862fb6)
2023-10-10T14:29:33.4426014Z     #13 Poco::(anonymous namespace)::RunnableHolder::run() /home/runner/work/RaftKeeper/RaftKeeper/contrib/poco/Foundation/src/Thread.cpp:56:11 (raftkeeper+0xcb9fef) (BuildId: 43558d85c161f514)
2023-10-10T14:29:33.4426877Z     #14 Poco::ThreadImpl::runnableEntry(void*) /home/runner/work/RaftKeeper/RaftKeeper/contrib/poco/Foundation/src/Thread_POSIX.cpp:358:27 (raftkeeper+0xcb86f6) (BuildId: 43558d85c161f514)
2023-10-10T14:29:33.4427327Z 
2023-10-10T14:29:33.4427553Z   Previous write of size 8 at 0x7b6000015968 by thread T110 (mutexes: write M2):
2023-10-10T14:29:33.4428247Z     #0 operator delete(void*, unsigned long) <null> (raftkeeper+0x7e055e) (BuildId: 43558d85c161f514)
2023-10-10T14:29:33.4428965Z     #1 RK::ConnectionHandler::destroyMe() /home/runner/work/RaftKeeper/RaftKeeper/src/Service/ConnectionHandler.cpp:722:5 (raftkeeper+0x88dcc5) (BuildId: 43558d85c161f514)
2023-10-10T14:29:33.4429889Z     #2 RK::ConnectionHandler::onSocketReadable(Poco::AutoPtr<RK::ReadableNotification> const&) /home/runner/work/RaftKeeper/RaftKeeper/src/Service/ConnectionHandler.cpp:216:21 (raftkeeper+0x88b1c2) (BuildId: 43558d85c161f514)
2023-10-10T14:29:33.4431079Z     #3 Poco::NObserver<RK::ConnectionHandler, RK::ReadableNotification>::notify(Poco::Notification*) const /home/runner/work/RaftKeeper/RaftKeeper/contrib/poco/Foundation/include/Poco/NObserver.h:86:5 (raftkeeper+0x897952) (BuildId: 43558d85c161f514)
2023-10-10T14:29:33.4432146Z     #4 Poco::NotificationCenter::postNotification(Poco::AutoPtr<Poco::Notification>) /home/runner/work/RaftKeeper/RaftKeeper/contrib/poco/Foundation/src/NotificationCenter.cpp:76:6 (raftkeeper+0xc6b447) (BuildId: 43558d85c161f514)
2023-10-10T14:29:33.4433113Z     #5 RK::SocketNotifier::dispatch(RK::SocketNotification*) /home/runner/work/RaftKeeper/RaftKeeper/src/Common/NIO/SocketNotifier.cpp:70:13 (raftkeeper+0x86a2ab) (BuildId: 43558d85c161f514)
2023-10-10T14:29:33.4434047Z     #6 RK::SocketReactor::dispatch(Poco::AutoPtr<RK::SocketNotifier>&, RK::SocketNotification*) /home/runner/work/RaftKeeper/RaftKeeper/src/Common/NIO/SocketReactor.cpp:295:20 (raftkeeper+0x8654d0) (BuildId: 43558d85c161f514)
2023-10-10T14:29:33.4435026Z     #7 RK::SocketReactor::dispatch(Poco::Net::Socket const&, RK::SocketNotification*) /home/runner/work/RaftKeeper/RaftKeeper/src/Common/NIO/SocketReactor.cpp:271:5 (raftkeeper+0x862fb6) (BuildId: 43558d85c161f514)
2023-10-10T14:29:33.4435851Z     #8 RK::SocketReactor::run() /home/runner/work/RaftKeeper/RaftKeeper/src/Common/NIO/SocketReactor.cpp:81:29 (raftkeeper+0x862fb6)
2023-10-10T14:29:33.4436626Z     #9 Poco::(anonymous namespace)::RunnableHolder::run() /home/runner/work/RaftKeeper/RaftKeeper/contrib/poco/Foundation/src/Thread.cpp:56:11 (raftkeeper+0xcb9fef) (BuildId: 43558d85c161f514)
2023-10-10T14:29:33.4437500Z     #10 Poco::ThreadImpl::runnableEntry(void*) /home/runner/work/RaftKeeper/RaftKeeper/contrib/poco/Foundation/src/Thread_POSIX.cpp:358:27 (raftkeeper+0xcb86f6) (BuildId: 43558d85c161f514)
2023-10-10T14:29:33.4437951Z 
2023-10-10T14:29:33.4438091Z   Mutex M0 (0x7b14000058e0) created at:
2023-10-10T14:29:33.4438508Z     #0 pthread_mutex_init <null> (raftkeeper+0x7613cf) (BuildId: 43558d85c161f514)
2023-10-10T14:29:33.4439169Z     #1 Poco::MutexImpl::MutexImpl() /home/runner/work/RaftKeeper/RaftKeeper/contrib/poco/Foundation/src/Mutex_POSIX.cpp:64:6 (raftkeeper+0xc6a75c) (BuildId: 43558d85c161f514)
2023-10-10T14:29:33.4439938Z     #2 Poco::Mutex::Mutex() /home/runner/work/RaftKeeper/RaftKeeper/contrib/poco/Foundation/src/Mutex.cpp:34:8 (raftkeeper+0xc6aac9) (BuildId: 43558d85c161f514)
2023-10-10T14:29:33.4441336Z     #3 Poco::Observer<RK::SvsSocketAcceptor<RK::ConnectionHandler, RK::SocketReactor>, RK::ReadableNotification>::Observer(Poco::Observer<RK::SvsSocketAcceptor<RK::ConnectionHandler, RK::SocketReactor>, RK::ReadableNotification> const&) /home/runner/work/RaftKeeper/RaftKeeper/contrib/poco/Foundation/include/Poco/Observer.h:53:2 (raftkeeper+0x7e87a2) (BuildId: 43558d85c161f514)
2023-10-10T14:29:33.4442671Z     #4 Poco::Observer<RK::SvsSocketAcceptor<RK::ConnectionHandler, RK::SocketReactor>, RK::ReadableNotification>::clone() const /home/runner/work/RaftKeeper/RaftKeeper/contrib/poco/Foundation/include/Poco/Observer.h:102:14 (raftkeeper+0x7e87a2)
2023-10-10T14:29:33.4443728Z     #5 Poco::NotificationCenter::addObserver(Poco::AbstractObserver const&) /home/runner/work/RaftKeeper/RaftKeeper/contrib/poco/Foundation/src/NotificationCenter.cpp:38:32 (raftkeeper+0xc6af29) (BuildId: 43558d85c161f514)
2023-10-10T14:29:33.4444899Z     #6 RK::SocketNotifier::addObserver(RK::SocketReactor*, Poco::AbstractObserver const&) /home/runner/work/RaftKeeper/RaftKeeper/src/Common/NIO/SocketNotifier.cpp:27:9 (raftkeeper+0x869a9e) (BuildId: 43558d85c161f514)
2023-10-10T14:29:33.4445903Z     #7 RK::SocketReactor::addEventHandler(Poco::Net::Socket const&, Poco::AbstractObserver const&) /home/runner/work/RaftKeeper/RaftKeeper/src/Common/NIO/SocketReactor.cpp:159:55 (raftkeeper+0x86405f) (BuildId: 43558d85c161f514)
2023-10-10T14:29:33.4447334Z     #8 RK::SvsSocketAcceptor<RK::ConnectionHandler, RK::SocketReactor>::SvsSocketAcceptor(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, RK::Context&, Poco::Net::ServerSocket&, RK::SocketReactor&, Poco::Timespan const&, unsigned int) /home/runner/work/RaftKeeper/RaftKeeper/src/Common/NIO/SvsSocketAcceptor.h:73:19 (raftkeeper+0x7e6f61) (BuildId: 43558d85c161f514)
2023-10-10T14:29:33.4449299Z     #9 void std::__1::allocator<RK::SvsSocketAcceptor<RK::ConnectionHandler, RK::SocketReactor> >::construct<RK::SvsSocketAcceptor<RK::ConnectionHandler, RK::SocketReactor>, char const (&) [8], RK::Context&, Poco::Net::ServerSocket&, RK::SvsSocketReactor<RK::SocketReactor>&, Poco::Timespan&, unsigned int&>(RK::SvsSocketAcceptor<RK::ConnectionHandler, RK::SocketReactor>*, char const (&) [8], RK::Context&, Poco::Net::ServerSocket&, RK::SvsSocketReactor<RK::SocketReactor>&, Poco::Timespan&, unsigned int&) /home/runner/work/RaftKeeper/RaftKeeper/contrib/libcxx/include/memory:886:28 (raftkeeper+0x7e6d79) (BuildId: 43558d85c161f514)
2023-10-10T14:29:33.4451923Z     #10 void std::__1::allocator_traits<std::__1::allocator<RK::SvsSocketAcceptor<RK::ConnectionHandler, RK::SocketReactor> > >::__construct<RK::SvsSocketAcceptor<RK::ConnectionHandler, RK::SocketReactor>, char const (&) [8], RK::Context&, Poco::Net::ServerSocket&, RK::SvsSocketReactor<RK::SocketReactor>&, Poco::Timespan&, unsigned int&>(std::__1::integral_constant<bool, true>, std::__1::allocator<RK::SvsSocketAcceptor<RK::ConnectionHandler, RK::SocketReactor> >&, RK::SvsSocketAcceptor<RK::ConnectionHandler, RK::SocketReactor>*, char const (&) [8], RK::Context&, Poco::Net::ServerSocket&, RK::SvsSocketReactor<RK::SocketReactor>&, Poco::Timespan&, unsigned int&) /home/runner/work/RaftKeeper/RaftKeeper/contrib/libcxx/include/__memory/allocator_traits.h:519:21 (raftkeeper+0x7e4c96) (BuildId: 43558d85c161f514)
2023-10-10T14:29:33.4454684Z     #11 void std::__1::allocator_traits<std::__1::allocator<RK::SvsSocketAcceptor<RK::ConnectionHandler, RK::SocketReactor> > >::construct<RK::SvsSocketAcceptor<RK::ConnectionHandler, RK::SocketReactor>, char const (&) [8], RK::Context&, Poco::Net::ServerSocket&, RK::SvsSocketReactor<RK::SocketReactor>&, Poco::Timespan&, unsigned int&>(std::__1::allocator<RK::SvsSocketAcceptor<RK::ConnectionHandler, RK::SocketReactor> >&, RK::SvsSocketAcceptor<RK::ConnectionHandler, RK::SocketReactor>*, char const (&) [8], RK::Context&, Poco::Net::ServerSocket&, RK::SvsSocketReactor<RK::SocketReactor>&, Poco::Timespan&, unsigned int&) /home/runner/work/RaftKeeper/RaftKeeper/contrib/libcxx/include/__memory/allocator_traits.h:481:14 (raftkeeper+0x7e4c96)
2023-10-10T14:29:33.4457237Z     #12 std::__1::__shared_ptr_emplace<RK::SvsSocketAcceptor<RK::ConnectionHandler, RK::SocketReactor>, std::__1::allocator<RK::SvsSocketAcceptor<RK::ConnectionHandler, RK::SocketReactor> > >::__shared_ptr_emplace<char const (&) [8], RK::Context&, Poco::Net::ServerSocket&, RK::SvsSocketReactor<RK::SocketReactor>&, Poco::Timespan&, unsigned int&>(std::__1::allocator<RK::SvsSocketAcceptor<RK::ConnectionHandler, RK::SocketReactor> >, char const (&) [8], RK::Context&, Poco::Net::ServerSocket&, RK::SvsSocketReactor<RK::SocketReactor>&, Poco::Timespan&, unsigned int&) /home/runner/work/RaftKeeper/RaftKeeper/contrib/libcxx/include/memory:2594:9 (raftkeeper+0x7e4c96)
2023-10-10T14:29:33.4459902Z     #13 std::__1::shared_ptr<RK::SvsSocketAcceptor<RK::ConnectionHandler, RK::SocketReactor> > std::__1::allocate_shared<RK::SvsSocketAcceptor<RK::ConnectionHandler, RK::SocketReactor>, std::__1::allocator<RK::SvsSocketAcceptor<RK::ConnectionHandler, RK::SocketReactor> >, char const (&) [8], RK::Context&, Poco::Net::ServerSocket&, RK::SvsSocketReactor<RK::SocketReactor>&, Poco::Timespan&, unsigned int&, void>(std::__1::allocator<RK::SvsSocketAcceptor<RK::ConnectionHandler, RK::SocketReactor> > const&, char const (&) [8], RK::Context&, Poco::Net::ServerSocket&, RK::SvsSocketReactor<RK::SocketReactor>&, Poco::Timespan&, unsigned int&) /home/runner/work/RaftKeeper/RaftKeeper/contrib/libcxx/include/memory:3360:55 (raftkeeper+0x7e4c96)
2023-10-10T14:29:33.4462312Z     #14 std::__1::shared_ptr<RK::SvsSocketAcceptor<RK::ConnectionHandler, RK::SocketReactor> > std::__1::make_shared<RK::SvsSocketAcceptor<RK::ConnectionHandler, RK::SocketReactor>, char const (&) [8], RK::Context&, Poco::Net::ServerSocket&, RK::SvsSocketReactor<RK::SocketReactor>&, Poco::Timespan&, unsigned int&, void>(char const (&) [8], RK::Context&, Poco::Net::ServerSocket&, RK::SvsSocketReactor<RK::SocketReactor>&, Poco::Timespan&, unsigned int&) /home/runner/work/RaftKeeper/RaftKeeper/contrib/libcxx/include/memory:3369:12 (raftkeeper+0x7e4c96)
2023-10-10T14:29:33.4463973Z     #15 RK::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&)::$_0::operator()(unsigned short) const /home/runner/work/RaftKeeper/RaftKeeper/programs/server/Server.cpp:174:29 (raftkeeper+0x7e4c96)
2023-10-10T14:29:33.4466059Z     #16 decltype(std::__1::forward<RK::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&)::$_0&>(fp)(std::__1::forward<unsigned short>(fp0))) std::__1::__invoke<RK::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&)::$_0&, unsigned short>(RK::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&)::$_0&, unsigned short&&) /home/runner/work/RaftKeeper/RaftKeeper/contrib/libcxx/include/type_traits:3676:1 (raftkeeper+0x7e4c96)
2023-10-10T14:29:33.4468498Z     #17 void std::__1::__invoke_void_return_wrapper<void>::__call<RK::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&)::$_0&, unsigned short>(RK::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&)::$_0&, unsigned short&&) /home/runner/work/RaftKeeper/RaftKeeper/contrib/libcxx/include/__functional_base:348:9 (raftkeeper+0x7e4c96)
2023-10-10T14:29:33.4470280Z     #18 std::__1::__function::__default_alloc_func<RK::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&)::$_0, void (unsigned short)>::operator()(unsigned short&&) /home/runner/work/RaftKeeper/RaftKeeper/contrib/libcxx/include/functional:1608:12 (raftkeeper+0x7e4c96)
2023-10-10T14:29:33.4472038Z     #19 void std::__1::__function::__policy_invoker<void (unsigned short)>::__call_impl<std::__1::__function::__default_alloc_func<RK::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&)::$_0, void (unsigned short)> >(std::__1::__function::__policy_storage const*, unsigned short) /home/runner/work/RaftKeeper/RaftKeeper/contrib/libcxx/include/functional:2089:16 (raftkeeper+0x7e4c96)
2023-10-10T14:29:33.4473390Z     #20 std::__1::__function::__policy_func<void (unsigned short)>::operator()(unsigned short&&) const /home/runner/work/RaftKeeper/RaftKeeper/contrib/libcxx/include/functional:2221:16 (raftkeeper+0x7e1cee) (BuildId: 43558d85c161f514)
2023-10-10T14:29:33.4474386Z     #21 std::__1::function<void (unsigned short)>::operator()(unsigned short) const /home/runner/work/RaftKeeper/RaftKeeper/contrib/libcxx/include/functional:2560:12 (raftkeeper+0x7e1cee)
2023-10-10T14:29:33.4475350Z     #22 RK::Server::createServer(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int, bool, std::__1::function<void (unsigned short)>&&) const /home/runner/work/RaftKeeper/RaftKeeper/programs/server/Server.cpp:51:9 (raftkeeper+0x7e1cee)
2023-10-10T14:29:33.4476546Z     #23 RK::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&) /home/runner/work/RaftKeeper/RaftKeeper/programs/server/Server.cpp:161:5 (raftkeeper+0x7e3059) (BuildId: 43558d85c161f514)
2023-10-10T14:29:33.4477557Z     #24 Poco::Util::Application::run() /home/runner/work/RaftKeeper/RaftKeeper/contrib/poco/Util/src/Application.cpp:358:8 (raftkeeper+0xb3cf54) (BuildId: 43558d85c161f514)
2023-10-10T14:29:33.4478324Z     #25 RK::Server::run() /home/runner/work/RaftKeeper/RaftKeeper/programs/server/Server.cpp:94:25 (raftkeeper+0x7e25d5) (BuildId: 43558d85c161f514)
2023-10-10T14:29:33.4479111Z     #26 Poco::Util::ServerApplication::run(int, char**) /home/runner/work/RaftKeeper/RaftKeeper/contrib/poco/Util/src/ServerApplication.cpp:585:9 (raftkeeper+0xb61ee0) (BuildId: 43558d85c161f514)
2023-10-10T14:29:33.4479973Z     #27 mainEntryRaftKeeperServer(int, char**) /home/runner/work/RaftKeeper/RaftKeeper/programs/server/Server.cpp:264:23 (raftkeeper+0x7e4993) (BuildId: 43558d85c161f514)
2023-10-10T14:29:33.4480702Z     #28 main /home/runner/work/RaftKeeper/RaftKeeper/programs/main.cpp:98:12 (raftkeeper+0x7e10ca) (BuildId: 43558d85c161f514)
2023-10-10T14:29:33.4481045Z 
2023-10-10T14:29:33.4481295Z   Mutex M1 (0x7b1c00000b68) created at:
2023-10-10T14:29:33.4481715Z     #0 pthread_mutex_init <null> (raftkeeper+0x7613cf) (BuildId: 43558d85c161f514)
2023-10-10T14:29:33.4482390Z     #1 Poco::MutexImpl::MutexImpl(bool) /home/runner/work/RaftKeeper/RaftKeeper/contrib/poco/Foundation/src/Mutex_POSIX.cpp:89:6 (raftkeeper+0xc6a84f) (BuildId: 43558d85c161f514)
2023-10-10T14:29:33.4483236Z     #2 Poco::FastMutexImpl::FastMutexImpl() /home/runner/work/RaftKeeper/RaftKeeper/contrib/poco/Foundation/src/Mutex_POSIX.cpp:165:33 (raftkeeper+0xc6ab0e) (BuildId: 43558d85c161f514)
2023-10-10T14:29:33.4484013Z     #3 Poco::FastMutex::FastMutex() /home/runner/work/RaftKeeper/RaftKeeper/contrib/poco/Foundation/src/Mutex.cpp:44:12 (raftkeeper+0xc6ab0e)
2023-10-10T14:29:33.4484758Z     #4 RK::PollSetImpl::PollSetImpl() /home/runner/work/RaftKeeper/RaftKeeper/src/Common/NIO/PollSet.cpp:40:5 (raftkeeper+0x866f38) (BuildId: 43558d85c161f514)
2023-10-10T14:29:33.4485490Z     #5 RK::PollSet::PollSet() /home/runner/work/RaftKeeper/RaftKeeper/src/Common/NIO/PollSet.cpp:442:16 (raftkeeper+0x866957) (BuildId: 43558d85c161f514)
2023-10-10T14:29:33.4486283Z     #6 RK::SocketReactor::SocketReactor(Poco::Timespan const&) /home/runner/work/RaftKeeper/RaftKeeper/src/Common/NIO/SocketReactor.cpp:37:16 (raftkeeper+0x8627cd) (BuildId: 43558d85c161f514)
2023-10-10T14:29:33.4487320Z     #7 RK::SvsSocketReactor<RK::SocketReactor>::SvsSocketReactor(Poco::Timespan const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) /home/runner/work/RaftKeeper/RaftKeeper/src/Common/NIO/SvsSocketReactor.h:33:9 (raftkeeper+0x7e69c7) (BuildId: 43558d85c161f514)
2023-10-10T14:29:33.4488512Z     #8 RK::SvsSocketAcceptor<RK::ConnectionHandler, RK::SocketReactor>::init() /home/runner/work/RaftKeeper/RaftKeeper/src/Common/NIO/SvsSocketAcceptor.h:200:37 (raftkeeper+0x7e716c) (BuildId: 43558d85c161f514)
2023-10-10T14:29:33.4489777Z     #9 RK::SvsSocketAcceptor<RK::ConnectionHandler, RK::SocketReactor>::SvsSocketAcceptor(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, RK::Context&, Poco::Net::ServerSocket&, RK::SocketReactor&, Poco::Timespan const&, unsigned int) /home/runner/work/RaftKeeper/RaftKeeper/src/Common/NIO/SvsSocketAcceptor.h:72:9 (raftkeeper+0x7e6f09) (BuildId: 43558d85c161f514)
2023-10-10T14:29:33.4535980Z     #10 void std::__1::allocator<RK::SvsSocketAcceptor<RK::ConnectionHandler, RK::SocketReactor> >::construct<RK::SvsSocketAcceptor<RK::ConnectionHandler, RK::SocketReactor>, char const (&) [8], RK::Context&, Poco::Net::ServerSocket&, RK::SvsSocketReactor<RK::SocketReactor>&, Poco::Timespan&, unsigned int&>(RK::SvsSocketAcceptor<RK::ConnectionHandler, RK::SocketReactor>*, char const (&) [8], RK::Context&, Poco::Net::ServerSocket&, RK::SvsSocketReactor<RK::SocketReactor>&, Poco::Timespan&, unsigned int&) /home/runner/work/RaftKeeper/RaftKeeper/contrib/libcxx/include/memory:886:28 (raftkeeper+0x7e6d79) (BuildId: 43558d85c161f514)
2023-10-10T14:29:33.4538632Z     #11 void std::__1::allocator_traits<std::__1::allocator<RK::SvsSocketAcceptor<RK::ConnectionHandler, RK::SocketReactor> > >::__construct<RK::SvsSocketAcceptor<RK::ConnectionHandler, RK::SocketReactor>, char const (&) [8], RK::Context&, Poco::Net::ServerSocket&, RK::SvsSocketReactor<RK::SocketReactor>&, Poco::Timespan&, unsigned int&>(std::__1::integral_constant<bool, true>, std::__1::allocator<RK::SvsSocketAcceptor<RK::ConnectionHandler, RK::SocketReactor> >&, RK::SvsSocketAcceptor<RK::ConnectionHandler, RK::SocketReactor>*, char const (&) [8], RK::Context&, Poco::Net::ServerSocket&, RK::SvsSocketReactor<RK::SocketReactor>&, Poco::Timespan&, unsigned int&) /home/runner/work/RaftKeeper/RaftKeeper/contrib/libcxx/include/__memory/allocator_traits.h:519:21 (raftkeeper+0x7e4c96) (BuildId: 43558d85c161f514)
2023-10-10T14:29:33.4541422Z     #12 void std::__1::allocator_traits<std::__1::allocator<RK::SvsSocketAcceptor<RK::ConnectionHandler, RK::SocketReactor> > >::construct<RK::SvsSocketAcceptor<RK::ConnectionHandler, RK::SocketReactor>, char const (&) [8], RK::Context&, Poco::Net::ServerSocket&, RK::SvsSocketReactor<RK::SocketReactor>&, Poco::Timespan&, unsigned int&>(std::__1::allocator<RK::SvsSocketAcceptor<RK::ConnectionHandler, RK::SocketReactor> >&, RK::SvsSocketAcceptor<RK::ConnectionHandler, RK::SocketReactor>*, char const (&) [8], RK::Context&, Poco::Net::ServerSocket&, RK::SvsSocketReactor<RK::SocketReactor>&, Poco::Timespan&, unsigned int&) /home/runner/work/RaftKeeper/RaftKeeper/contrib/libcxx/include/__memory/allocator_traits.h:481:14 (raftkeeper+0x7e4c96)
2023-10-10T14:29:33.4543982Z     #13 std::__1::__shared_ptr_emplace<RK::SvsSocketAcceptor<RK::ConnectionHandler, RK::SocketReactor>, std::__1::allocator<RK::SvsSocketAcceptor<RK::ConnectionHandler, RK::SocketReactor> > >::__shared_ptr_emplace<char const (&) [8], RK::Context&, Poco::Net::ServerSocket&, RK::SvsSocketReactor<RK::SocketReactor>&, Poco::Timespan&, unsigned int&>(std::__1::allocator<RK::SvsSocketAcceptor<RK::ConnectionHandler, RK::SocketReactor> >, char const (&) [8], RK::Context&, Poco::Net::ServerSocket&, RK::SvsSocketReactor<RK::SocketReactor>&, Poco::Timespan&, unsigned int&) /home/runner/work/RaftKeeper/RaftKeeper/contrib/libcxx/include/memory:2594:9 (raftkeeper+0x7e4c96)
2023-10-10T14:29:33.4546728Z     #14 std::__1::shared_ptr<RK::SvsSocketAcceptor<RK::ConnectionHandler, RK::SocketReactor> > std::__1::allocate_shared<RK::SvsSocketAcceptor<RK::ConnectionHandler, RK::SocketReactor>, std::__1::allocator<RK::SvsSocketAcceptor<RK::ConnectionHandler, RK::SocketReactor> >, char const (&) [8], RK::Context&, Poco::Net::ServerSocket&, RK::SvsSocketReactor<RK::SocketReactor>&, Poco::Timespan&, unsigned int&, void>(std::__1::allocator<RK::SvsSocketAcceptor<RK::ConnectionHandler, RK::SocketReactor> > const&, char const (&) [8], RK::Context&, Poco::Net::ServerSocket&, RK::SvsSocketReactor<RK::SocketReactor>&, Poco::Timespan&, unsigned int&) /home/runner/work/RaftKeeper/RaftKeeper/contrib/libcxx/include/memory:3360:55 (raftkeeper+0x7e4c96)
2023-10-10T14:29:33.4554359Z     #15 std::__1::shared_ptr<RK::SvsSocketAcceptor<RK::ConnectionHandler, RK::SocketReactor> > std::__1::make_shared<RK::SvsSocketAcceptor<RK::ConnectionHandler, RK::SocketReactor>, char const (&) [8], RK::Context&, Poco::Net::ServerSocket&, RK::SvsSocketReactor<RK::SocketReactor>&, Poco::Timespan&, unsigned int&, void>(char const (&) [8], RK::Context&, Poco::Net::ServerSocket&, RK::SvsSocketReactor<RK::SocketReactor>&, Poco::Timespan&, unsigned int&) /home/runner/work/RaftKeeper/RaftKeeper/contrib/libcxx/include/memory:3369:12 (raftkeeper+0x7e4c96)
2023-10-10T14:29:33.4556204Z     #16 RK::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&)::$_0::operator()(unsigned short) const /home/runner/work/RaftKeeper/RaftKeeper/programs/server/Server.cpp:174:29 (raftkeeper+0x7e4c96)
2023-10-10T14:29:33.4558307Z     #17 decltype(std::__1::forward<RK::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&)::$_0&>(fp)(std::__1::forward<unsigned short>(fp0))) std::__1::__invoke<RK::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&)::$_0&, unsigned short>(RK::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&)::$_0&, unsigned short&&) /home/runner/work/RaftKeeper/RaftKeeper/contrib/libcxx/include/type_traits:3676:1 (raftkeeper+0x7e4c96)
2023-10-10T14:29:33.4560701Z     #18 void std::__1::__invoke_void_return_wrapper<void>::__call<RK::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&)::$_0&, unsigned short>(RK::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&)::$_0&, unsigned short&&) /home/runner/work/RaftKeeper/RaftKeeper/contrib/libcxx/include/__functional_base:348:9 (raftkeeper+0x7e4c96)
2023-10-10T14:29:33.4562758Z     #19 std::__1::__function::__default_alloc_func<RK::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&)::$_0, void (unsigned short)>::operator()(unsigned short&&) /home/runner/work/RaftKeeper/RaftKeeper/contrib/libcxx/include/functional:1608:12 (raftkeeper+0x7e4c96)
2023-10-10T14:29:33.4564406Z     #20 void std::__1::__function::__policy_invoker<void (unsigned short)>::__call_impl<std::__1::__function::__default_alloc_func<RK::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&)::$_0, void (unsigned short)> >(std::__1::__function::__policy_storage const*, unsigned short) /home/runner/work/RaftKeeper/RaftKeeper/contrib/libcxx/include/functional:2089:16 (raftkeeper+0x7e4c96)
2023-10-10T14:29:33.4565917Z     #21 std::__1::__function::__policy_func<void (unsigned short)>::operator()(unsigned short&&) const /home/runner/work/RaftKeeper/RaftKeeper/contrib/libcxx/include/functional:2221:16 (raftkeeper+0x7e1cee) (BuildId: 43558d85c161f514)
2023-10-10T14:29:33.4566821Z     #22 std::__1::function<void (unsigned short)>::operator()(unsigned short) const /home/runner/work/RaftKeeper/RaftKeeper/contrib/libcxx/include/functional:2560:12 (raftkeeper+0x7e1cee)
2023-10-10T14:29:33.4567922Z     #23 RK::Server::createServer(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int, bool, std::__1::function<void (unsigned short)>&&) const /home/runner/work/RaftKeeper/RaftKeeper/programs/server/Server.cpp:51:9 (raftkeeper+0x7e1cee)
2023-10-10T14:29:33.4569106Z     #24 RK::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&) /home/runner/work/RaftKeeper/RaftKeeper/programs/server/Server.cpp:161:5 (raftkeeper+0x7e3059) (BuildId: 43558d85c161f514)
2023-10-10T14:29:33.4570136Z     #25 Poco::Util::Application::run() /home/runner/work/RaftKeeper/RaftKeeper/contrib/poco/Util/src/Application.cpp:358:8 (raftkeeper+0xb3cf54) (BuildId: 43558d85c161f514)
2023-10-10T14:29:33.4570893Z     #26 RK::Server::run() /home/runner/work/RaftKeeper/RaftKeeper/programs/server/Server.cpp:94:25 (raftkeeper+0x7e25d5) (BuildId: 43558d85c161f514)
2023-10-10T14:29:33.4572162Z     #27 Poco::Util::ServerApplication::run(int, char**) /home/runner/work/RaftKeeper/RaftKeeper/contrib/poco/Util/src/ServerApplication.cpp:585:9 (raftkeeper+0xb61ee0) (BuildId: 43558d85c161f514)
2023-10-10T14:29:33.4573019Z     #28 mainEntryRaftKeeperServer(int, char**) /home/runner/work/RaftKeeper/RaftKeeper/programs/server/Server.cpp:264:23 (raftkeeper+0x7e4993) (BuildId: 43558d85c161f514)
2023-10-10T14:29:33.4573757Z     #29 main /home/runner/work/RaftKeeper/RaftKeeper/programs/main.cpp:98:12 (raftkeeper+0x7e10ca) (BuildId: 43558d85c161f514)
2023-10-10T14:29:33.4574106Z 
2023-10-10T14:29:33.4574250Z   Mutex M2 (0x7b1400072f80) created at:
2023-10-10T14:29:33.4574665Z     #0 pthread_mutex_init <null> (raftkeeper+0x7613cf) (BuildId: 43558d85c161f514)
2023-10-10T14:29:33.4575337Z     #1 Poco::MutexImpl::MutexImpl() /home/runner/work/RaftKeeper/RaftKeeper/contrib/poco/Foundation/src/Mutex_POSIX.cpp:64:6 (raftkeeper+0xc6a75c) (BuildId: 43558d85c161f514)
2023-10-10T14:29:33.4576121Z     #2 Poco::Mutex::Mutex() /home/runner/work/RaftKeeper/RaftKeeper/contrib/poco/Foundation/src/Mutex.cpp:34:8 (raftkeeper+0xc6aac9) (BuildId: 43558d85c161f514)
2023-10-10T14:29:33.4577155Z     #3 Poco::NObserver<RK::ConnectionHandler, RK::ReadableNotification>::NObserver(Poco::NObserver<RK::ConnectionHandler, RK::ReadableNotification> const&) /home/runner/work/RaftKeeper/RaftKeeper/contrib/poco/Foundation/include/Poco/NObserver.h:55:2 (raftkeeper+0x897c42) (BuildId: 43558d85c161f514)
2023-10-10T14:29:33.4578277Z     #4 Poco::NObserver<RK::ConnectionHandler, RK::ReadableNotification>::clone() const /home/runner/work/RaftKeeper/RaftKeeper/contrib/poco/Foundation/include/Poco/NObserver.h:104:14 (raftkeeper+0x897c42)
2023-10-10T14:29:33.4579268Z     #5 Poco::NotificationCenter::addObserver(Poco::AbstractObserver const&) /home/runner/work/RaftKeeper/RaftKeeper/contrib/poco/Foundation/src/NotificationCenter.cpp:38:32 (raftkeeper+0xc6af29) (BuildId: 43558d85c161f514)
2023-10-10T14:29:33.4580274Z     #6 RK::SocketNotifier::addObserver(RK::SocketReactor*, Poco::AbstractObserver const&) /home/runner/work/RaftKeeper/RaftKeeper/src/Common/NIO/SocketNotifier.cpp:27:9 (raftkeeper+0x869a9e) (BuildId: 43558d85c161f514)
2023-10-10T14:29:33.4581365Z     #7 RK::SocketReactor::addEventHandlers(Poco::Net::Socket const&, std::__1::vector<Poco::AbstractObserver*, std::__1::allocator<Poco::AbstractObserver*> > const&) /home/runner/work/RaftKeeper/RaftKeeper/src/Common/NIO/SocketReactor.cpp:174:60 (raftkeeper+0x8647b5) (BuildId: 43558d85c161f514)
2023-10-10T14:29:33.4582613Z     #8 RK::ConnectionHandler::ConnectionHandler(RK::Context&, Poco::Net::StreamSocket&, RK::SocketReactor&) /home/runner/work/RaftKeeper/RaftKeeper/src/Service/ConnectionHandler.cpp:90:13 (raftkeeper+0x889386) (BuildId: 43558d85c161f514)
2023-10-10T14:29:33.4592660Z     #9 RK::SvsSocketAcceptor<RK::ConnectionHandler, RK::SocketReactor>::createServiceHandler(Poco::Net::StreamSocket&) /home/runner/work/RaftKeeper/RaftKeeper/src/Common/NIO/SvsSocketAcceptor.h:161:25 (raftkeeper+0x7e7cc5) (BuildId: 43558d85c161f514)
2023-10-10T14:29:33.4593989Z     #10 RK::SvsSocketAcceptor<RK::ConnectionHandler, RK::SocketReactor>::onAccept(RK::ReadableNotification*) /home/runner/work/RaftKeeper/RaftKeeper/src/Common/NIO/SvsSocketAcceptor.h:138:9 (raftkeeper+0x7e7484) (BuildId: 43558d85c161f514)
2023-10-10T14:29:33.4595158Z     #11 Poco::Observer<RK::SvsSocketAcceptor<RK::ConnectionHandler, RK::SocketReactor>, RK::ReadableNotification>::notify(Poco::Notification*) const /home/runner/work/RaftKeeper/RaftKeeper/contrib/poco/Foundation/include/Poco/Observer.h:84:5 (raftkeeper+0x7e850e) (BuildId: 43558d85c161f514)
2023-10-10T14:29:33.4596307Z     #12 Poco::NotificationCenter::postNotification(Poco::AutoPtr<Poco::Notification>) /home/runner/work/RaftKeeper/RaftKeeper/contrib/poco/Foundation/src/NotificationCenter.cpp:76:6 (raftkeeper+0xc6b447) (BuildId: 43558d85c161f514)
2023-10-10T14:29:33.4597284Z     #13 RK::SocketNotifier::dispatch(RK::SocketNotification*) /home/runner/work/RaftKeeper/RaftKeeper/src/Common/NIO/SocketNotifier.cpp:70:13 (raftkeeper+0x86a2ab) (BuildId: 43558d85c161f514)
2023-10-10T14:29:33.4598235Z     #14 RK::SocketReactor::dispatch(Poco::AutoPtr<RK::SocketNotifier>&, RK::SocketNotification*) /home/runner/work/RaftKeeper/RaftKeeper/src/Common/NIO/SocketReactor.cpp:295:20 (raftkeeper+0x8654d0) (BuildId: 43558d85c161f514)
2023-10-10T14:29:33.4599208Z     #15 RK::SocketReactor::dispatch(Poco::Net::Socket const&, RK::SocketNotification*) /home/runner/work/RaftKeeper/RaftKeeper/src/Common/NIO/SocketReactor.cpp:271:5 (raftkeeper+0x862fb6) (BuildId: 43558d85c161f514)
2023-10-10T14:29:33.4600024Z     #16 RK::SocketReactor::run() /home/runner/work/RaftKeeper/RaftKeeper/src/Common/NIO/SocketReactor.cpp:81:29 (raftkeeper+0x862fb6)
2023-10-10T14:29:33.4600813Z     #17 Poco::(anonymous namespace)::RunnableHolder::run() /home/runner/work/RaftKeeper/RaftKeeper/contrib/poco/Foundation/src/Thread.cpp:56:11 (raftkeeper+0xcb9fef) (BuildId: 43558d85c161f514)
2023-10-10T14:29:33.4601838Z     #18 Poco::ThreadImpl::runnableEntry(void*) /home/runner/work/RaftKeeper/RaftKeeper/contrib/poco/Foundation/src/Thread_POSIX.cpp:358:27 (raftkeeper+0xcb86f6) (BuildId: 43558d85c161f514)
2023-10-10T14:29:33.4602288Z 
2023-10-10T14:29:33.4602725Z   Thread T108 '#2' (tid=119, running) created by main thread at:
2023-10-10T14:29:33.4603192Z     #0 pthread_create <null> (raftkeeper+0x75fbcd) (BuildId: 43558d85c161f514)
2023-10-10T14:29:33.4604054Z     #1 Poco::ThreadImpl::startImpl(Poco::SharedPtr<Poco::Runnable, Poco::ReferenceCounter, Poco::ReleasePolicy<Poco::Runnable> >) /home/runner/work/RaftKeeper/RaftKeeper/contrib/poco/Foundation/src/Thread_POSIX.cpp:206:7 (raftkeeper+0xcb7f46) (BuildId: 43558d85c161f514)
2023-10-10T14:29:33.4605047Z     #2 Poco::Thread::start(Poco::Runnable&) /home/runner/work/RaftKeeper/RaftKeeper/contrib/poco/Foundation/src/Thread.cpp:129:2 (raftkeeper+0xcb990c) (BuildId: 43558d85c161f514)
2023-10-10T14:29:33.4606078Z     #3 RK::SvsSocketReactor<RK::SocketReactor>::SvsSocketReactor(Poco::Timespan const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) /home/runner/work/RaftKeeper/RaftKeeper/src/Common/NIO/SvsSocketReactor.h:35:17 (raftkeeper+0x7e69f5) (BuildId: 43558d85c161f514)
2023-10-10T14:29:33.4607538Z     #4 void std::__1::allocator<RK::SvsSocketReactor<RK::SocketReactor> >::construct<RK::SvsSocketReactor<RK::SocketReactor>, Poco::Timespan&, char const (&) [9]>(RK::SvsSocketReactor<RK::SocketReactor>*, Poco::Timespan&, char const (&) [9]) /home/runner/work/RaftKeeper/RaftKeeper/contrib/libcxx/include/memory:886:28 (raftkeeper+0x7e6927) (BuildId: 43558d85c161f514)
2023-10-10T14:29:33.4609206Z     #5 void std::__1::allocator_traits<std::__1::allocator<RK::SvsSocketReactor<RK::SocketReactor> > >::__construct<RK::SvsSocketReactor<RK::SocketReactor>, Poco::Timespan&, char const (&) [9]>(std::__1::integral_constant<bool, true>, std::__1::allocator<RK::SvsSocketReactor<RK::SocketReactor> >&, RK::SvsSocketReactor<RK::SocketReactor>*, Poco::Timespan&, char const (&) [9]) /home/runner/work/RaftKeeper/RaftKeeper/contrib/libcxx/include/__memory/allocator_traits.h:519:21 (raftkeeper+0x7e4b8b) (BuildId: 43558d85c161f514)
2023-10-10T14:29:33.4611120Z     #6 void std::__1::allocator_traits<std::__1::allocator<RK::SvsSocketReactor<RK::SocketReactor> > >::construct<RK::SvsSocketReactor<RK::SocketReactor>, Poco::Timespan&, char const (&) [9]>(std::__1::allocator<RK::SvsSocketReactor<RK::SocketReactor> >&, RK::SvsSocketReactor<RK::SocketReactor>*, Poco::Timespan&, char const (&) [9]) /home/runner/work/RaftKeeper/RaftKeeper/contrib/libcxx/include/__memory/allocator_traits.h:481:14 (raftkeeper+0x7e4b8b)
2023-10-10T14:29:33.4612700Z     #7 std::__1::__shared_ptr_emplace<RK::SvsSocketReactor<RK::SocketReactor>, std::__1::allocator<RK::SvsSocketReactor<RK::SocketReactor> > >::__shared_ptr_emplace<Poco::Timespan&, char const (&) [9]>(std::__1::allocator<RK::SvsSocketReactor<RK::SocketReactor> >, Poco::Timespan&, char const (&) [9]) /home/runner/work/RaftKeeper/RaftKeeper/contrib/libcxx/include/memory:2594:9 (raftkeeper+0x7e4b8b)
2023-10-10T14:29:33.4614300Z     #8 std::__1::shared_ptr<RK::SvsSocketReactor<RK::SocketReactor> > std::__1::allocate_shared<RK::SvsSocketReactor<RK::SocketReactor>, std::__1::allocator<RK::SvsSocketReactor<RK::SocketReactor> >, Poco::Timespan&, char const (&) [9], void>(std::__1::allocator<RK::SvsSocketReactor<RK::SocketReactor> > const&, Poco::Timespan&, char const (&) [9]) /home/runner/work/RaftKeeper/RaftKeeper/contrib/libcxx/include/memory:3360:55 (raftkeeper+0x7e4b8b)
2023-10-10T14:29:33.4615730Z     #9 std::__1::shared_ptr<RK::SvsSocketReactor<RK::SocketReactor> > std::__1::make_shared<RK::SvsSocketReactor<RK::SocketReactor>, Poco::Timespan&, char const (&) [9], void>(Poco::Timespan&, char const (&) [9]) /home/runner/work/RaftKeeper/RaftKeeper/contrib/libcxx/include/memory:3369:12 (raftkeeper+0x7e4b8b)
2023-10-10T14:29:33.4617012Z     #10 RK::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&)::$_0::operator()(unsigned short) const /home/runner/work/RaftKeeper/RaftKeeper/programs/server/Server.cpp:171:22 (raftkeeper+0x7e4b8b)
2023-10-10T14:29:33.4619083Z     #11 decltype(std::__1::forward<RK::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&)::$_0&>(fp)(std::__1::forward<unsigned short>(fp0))) std::__1::__invoke<RK::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&)::$_0&, unsigned short>(RK::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&)::$_0&, unsigned short&&) /home/runner/work/RaftKeeper/RaftKeeper/contrib/libcxx/include/type_traits:3676:1 (raftkeeper+0x7e4b8b)
2023-10-10T14:29:33.4621633Z     #12 void std::__1::__invoke_void_return_wrapper<void>::__call<RK::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&)::$_0&, unsigned short>(RK::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&)::$_0&, unsigned short&&) /home/runner/work/RaftKeeper/RaftKeeper/contrib/libcxx/include/__functional_base:348:9 (raftkeeper+0x7e4b8b)
2023-10-10T14:29:33.4623504Z     #13 std::__1::__function::__default_alloc_func<RK::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&)::$_0, void (unsigned short)>::operator()(unsigned short&&) /home/runner/work/RaftKeeper/RaftKeeper/contrib/libcxx/include/functional:1608:12 (raftkeeper+0x7e4b8b)
2023-10-10T14:29:33.4625126Z     #14 void std::__1::__function::__policy_invoker<void (unsigned short)>::__call_impl<std::__1::__function::__default_alloc_func<RK::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&)::$_0, void (unsigned short)> >(std::__1::__function::__policy_storage const*, unsigned short) /home/runner/work/RaftKeeper/RaftKeeper/contrib/libcxx/include/functional:2089:16 (raftkeeper+0x7e4b8b)
2023-10-10T14:29:33.4626454Z     #15 std::__1::__function::__policy_func<void (unsigned short)>::operator()(unsigned short&&) const /home/runner/work/RaftKeeper/RaftKeeper/contrib/libcxx/include/functional:2221:16 (raftkeeper+0x7e1cee) (BuildId: 43558d85c161f514)
2023-10-10T14:29:33.4627342Z     #16 std::__1::function<void (unsigned short)>::operator()(unsigned short) const /home/runner/work/RaftKeeper/RaftKeeper/contrib/libcxx/include/functional:2560:12 (raftkeeper+0x7e1cee)
2023-10-10T14:29:33.4628317Z     #17 RK::Server::createServer(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int, bool, std::__1::function<void (unsigned short)>&&) const /home/runner/work/RaftKeeper/RaftKeeper/programs/server/Server.cpp:51:9 (raftkeeper+0x7e1cee)
2023-10-10T14:29:33.4629507Z     #18 RK::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&) /home/runner/work/RaftKeeper/RaftKeeper/programs/server/Server.cpp:161:5 (raftkeeper+0x7e3059) (BuildId: 43558d85c161f514)
2023-10-10T14:29:33.4630536Z     #19 Poco::Util::Application::run() /home/runner/work/RaftKeeper/RaftKeeper/contrib/poco/Util/src/Application.cpp:358:8 (raftkeeper+0xb3cf54) (BuildId: 43558d85c161f514)
2023-10-10T14:29:33.4631296Z     #20 RK::Server::run() /home/runner/work/RaftKeeper/RaftKeeper/programs/server/Server.cpp:94:25 (raftkeeper+0x7e25d5) (BuildId: 43558d85c161f514)
2023-10-10T14:29:33.4632089Z     #21 Poco::Util::ServerApplication::run(int, char**) /home/runner/work/RaftKeeper/RaftKeeper/contrib/poco/Util/src/ServerApplication.cpp:585:9 (raftkeeper+0xb61ee0) (BuildId: 43558d85c161f514)
2023-10-10T14:29:33.4632948Z     #22 mainEntryRaftKeeperServer(int, char**) /home/runner/work/RaftKeeper/RaftKeeper/programs/server/Server.cpp:264:23 (raftkeeper+0x7e4993) (BuildId: 43558d85c161f514)
2023-10-10T14:29:33.4633680Z     #23 main /home/runner/work/RaftKeeper/RaftKeeper/programs/main.cpp:98:12 (raftkeeper+0x7e10ca) (BuildId: 43558d85c161f514)
2023-10-10T14:29:33.4634028Z 
2023-10-10T14:29:33.4634357Z   Thread T110 'IO-Hdlr#1' (tid=121, running) created by main thread at:
2023-10-10T14:29:33.4634821Z     #0 pthread_create <null> (raftkeeper+0x75fbcd) (BuildId: 43558d85c161f514)
2023-10-10T14:29:33.4635786Z     #1 Poco::ThreadImpl::startImpl(Poco::SharedPtr<Poco::Runnable, Poco::ReferenceCounter, Poco::ReleasePolicy<Poco::Runnable> >) /home/runner/work/RaftKeeper/RaftKeeper/contrib/poco/Foundation/src/Thread_POSIX.cpp:206:7 (raftkeeper+0xcb7f46) (BuildId: 43558d85c161f514)
2023-10-10T14:29:33.4636777Z     #2 Poco::Thread::start(Poco::Runnable&) /home/runner/work/RaftKeeper/RaftKeeper/contrib/poco/Foundation/src/Thread.cpp:129:2 (raftkeeper+0xcb990c) (BuildId: 43558d85c161f514)
2023-10-10T14:29:33.4637931Z     #3 RK::SvsSocketReactor<RK::SocketReactor>::SvsSocketReactor(Poco::Timespan const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) /home/runner/work/RaftKeeper/RaftKeeper/src/Common/NIO/SvsSocketReactor.h:35:17 (raftkeeper+0x7e69f5) (BuildId: 43558d85c161f514)
2023-10-10T14:29:33.4639017Z     #4 RK::SvsSocketAcceptor<RK::ConnectionHandler, RK::SocketReactor>::init() /home/runner/work/RaftKeeper/RaftKeeper/src/Common/NIO/SvsSocketAcceptor.h:200:37 (raftkeeper+0x7e716c) (BuildId: 43558d85c161f514)
2023-10-10T14:29:33.4640293Z     #5 RK::SvsSocketAcceptor<RK::ConnectionHandler, RK::SocketReactor>::SvsSocketAcceptor(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, RK::Context&, Poco::Net::ServerSocket&, RK::SocketReactor&, Poco::Timespan const&, unsigned int) /home/runner/work/RaftKeeper/RaftKeeper/src/Common/NIO/SvsSocketAcceptor.h:72:9 (raftkeeper+0x7e6f09) (BuildId: 43558d85c161f514)
2023-10-10T14:29:33.4643030Z     #6 void std::__1::allocator<RK::SvsSocketAcceptor<RK::ConnectionHandler, RK::SocketReactor> >::construct<RK::SvsSocketAcceptor<RK::ConnectionHandler, RK::SocketReactor>, char const (&) [8], RK::Context&, Poco::Net::ServerSocket&, RK::SvsSocketReactor<RK::SocketReactor>&, Poco::Timespan&, unsigned int&>(RK::SvsSocketAcceptor<RK::ConnectionHandler, RK::SocketReactor>*, char const (&) [8], RK::Context&, Poco::Net::ServerSocket&, RK::SvsSocketReactor<RK::SocketReactor>&, Poco::Timespan&, unsigned int&) /home/runner/work/RaftKeeper/RaftKeeper/contrib/libcxx/include/memory:886:28 (raftkeeper+0x7e6d79) (BuildId: 43558d85c161f514)
2023-10-10T14:29:33.4645650Z     #7 void std::__1::allocator_traits<std::__1::allocator<RK::SvsSocketAcceptor<RK::ConnectionHandler, RK::SocketReactor> > >::__construct<RK::SvsSocketAcceptor<RK::ConnectionHandler, RK::SocketReactor>, char const (&) [8], RK::Context&, Poco::Net::ServerSocket&, RK::SvsSocketReactor<RK::SocketReactor>&, Poco::Timespan&, unsigned int&>(std::__1::integral_constant<bool, true>, std::__1::allocator<RK::SvsSocketAcceptor<RK::ConnectionHandler, RK::SocketReactor> >&, RK::SvsSocketAcceptor<RK::ConnectionHandler, RK::SocketReactor>*, char const (&) [8], RK::Context&, Poco::Net::ServerSocket&, RK::SvsSocketReactor<RK::SocketReactor>&, Poco::Timespan&, unsigned int&) /home/runner/work/RaftKeeper/RaftKeeper/contrib/libcxx/include/__memory/allocator_traits.h:519:21 (raftkeeper+0x7e4c96) (BuildId: 43558d85c161f514)
2023-10-10T14:29:33.4648438Z     #8 void std::__1::allocator_traits<std::__1::allocator<RK::SvsSocketAcceptor<RK::ConnectionHandler, RK::SocketReactor> > >::construct<RK::SvsSocketAcceptor<RK::ConnectionHandler, RK::SocketReactor>, char const (&) [8], RK::Context&, Poco::Net::ServerSocket&, RK::SvsSocketReactor<RK::SocketReactor>&, Poco::Timespan&, unsigned int&>(std::__1::allocator<RK::SvsSocketAcceptor<RK::ConnectionHandler, RK::SocketReactor> >&, RK::SvsSocketAcceptor<RK::ConnectionHandler, RK::SocketReactor>*, char const (&) [8], RK::Context&, Poco::Net::ServerSocket&, RK::SvsSocketReactor<RK::SocketReactor>&, Poco::Timespan&, unsigned int&) /home/runner/work/RaftKeeper/RaftKeeper/contrib/libcxx/include/__memory/allocator_traits.h:481:14 (raftkeeper+0x7e4c96)
2023-10-10T14:29:33.4651135Z     #9 std::__1::__shared_ptr_emplace<RK::SvsSocketAcceptor<RK::ConnectionHandler, RK::SocketReactor>, std::__1::allocator<RK::SvsSocketAcceptor<RK::ConnectionHandler, RK::SocketReactor> > >::__shared_ptr_emplace<char const (&) [8], RK::Context&, Poco::Net::ServerSocket&, RK::SvsSocketReactor<RK::SocketReactor>&, Poco::Timespan&, unsigned int&>(std::__1::allocator<RK::SvsSocketAcceptor<RK::ConnectionHandler, RK::SocketReactor> >, char const (&) [8], RK::Context&, Poco::Net::ServerSocket&, RK::SvsSocketReactor<RK::SocketReactor>&, Poco::Timespan&, unsigned int&) /home/runner/work/RaftKeeper/RaftKeeper/contrib/libcxx/include/memory:2594:9 (raftkeeper+0x7e4c96)
2023-10-10T14:29:33.4653674Z     #10 std::__1::shared_ptr<RK::SvsSocketAcceptor<RK::ConnectionHandler, RK::SocketReactor> > std::__1::allocate_shared<RK::SvsSocketAcceptor<RK::ConnectionHandler, RK::SocketReactor>, std::__1::allocator<RK::SvsSocketAcceptor<RK::ConnectionHandler, RK::SocketReactor> >, char const (&) [8], RK::Context&, Poco::Net::ServerSocket&, RK::SvsSocketReactor<RK::SocketReactor>&, Poco::Timespan&, unsigned int&, void>(std::__1::allocator<RK::SvsSocketAcceptor<RK::ConnectionHandler, RK::SocketReactor> > const&, char const (&) [8], RK::Context&, Poco::Net::ServerSocket&, RK::SvsSocketReactor<RK::SocketReactor>&, Poco::Timespan&, unsigned int&) /home/runner/work/RaftKeeper/RaftKeeper/contrib/libcxx/include/memory:3360:55 (raftkeeper+0x7e4c96)
2023-10-10T14:29:33.4656144Z     #11 std::__1::shared_ptr<RK::SvsSocketAcceptor<RK::ConnectionHandler, RK::SocketReactor> > std::__1::make_shared<RK::SvsSocketAcceptor<RK::ConnectionHandler, RK::SocketReactor>, char const (&) [8], RK::Context&, Poco::Net::ServerSocket&, RK::SvsSocketReactor<RK::SocketReactor>&, Poco::Timespan&, unsigned int&, void>(char const (&) [8], RK::Context&, Poco::Net::ServerSocket&, RK::SvsSocketReactor<RK::SocketReactor>&, Poco::Timespan&, unsigned int&) /home/runner/work/RaftKeeper/RaftKeeper/contrib/libcxx/include/memory:3369:12 (raftkeeper+0x7e4c96)
2023-10-10T14:29:33.4657826Z     #12 RK::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&)::$_0::operator()(unsigned short) const /home/runner/work/RaftKeeper/RaftKeeper/programs/server/Server.cpp:174:29 (raftkeeper+0x7e4c96)
2023-10-10T14:29:33.4659903Z     #13 decltype(std::__1::forward<RK::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&)::$_0&>(fp)(std::__1::forward<unsigned short>(fp0))) std::__1::__invoke<RK::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&)::$_0&, unsigned short>(RK::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&)::$_0&, unsigned short&&) /home/runner/work/RaftKeeper/RaftKeeper/contrib/libcxx/include/type_traits:3676:1 (raftkeeper+0x7e4c96)
2023-10-10T14:29:33.4662329Z     #14 void std::__1::__invoke_void_return_wrapper<void>::__call<RK::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&)::$_0&, unsigned short>(RK::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&)::$_0&, unsigned short&&) /home/runner/work/RaftKeeper/RaftKeeper/contrib/libcxx/include/__functional_base:348:9 (raftkeeper+0x7e4c96)
2023-10-10T14:29:33.4664211Z     #15 std::__1::__function::__default_alloc_func<RK::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&)::$_0, void (unsigned short)>::operator()(unsigned short&&) /home/runner/work/RaftKeeper/RaftKeeper/contrib/libcxx/include/functional:1608:12 (raftkeeper+0x7e4c96)
2023-10-10T14:29:33.4665855Z     #16 void std::__1::__function::__policy_invoker<void (unsigned short)>::__call_impl<std::__1::__function::__default_alloc_func<RK::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&)::$_0, void (unsigned short)> >(std::__1::__function::__policy_storage const*, unsigned short) /home/runner/work/RaftKeeper/RaftKeeper/contrib/libcxx/include/functional:2089:16 (raftkeeper+0x7e4c96)
2023-10-10T14:29:33.4667318Z     #17 std::__1::__function::__policy_func<void (unsigned short)>::operator()(unsigned short&&) const /home/runner/work/RaftKeeper/RaftKeeper/contrib/libcxx/include/functional:2221:16 (raftkeeper+0x7e1cee) (BuildId: 43558d85c161f514)
2023-10-10T14:29:33.4668218Z     #18 std::__1::function<void (unsigned short)>::operator()(unsigned short) const /home/runner/work/RaftKeeper/RaftKeeper/contrib/libcxx/include/functional:2560:12 (raftkeeper+0x7e1cee)
2023-10-10T14:29:33.4669206Z     #19 RK::Server::createServer(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int, bool, std::__1::function<void (unsigned short)>&&) const /home/runner/work/RaftKeeper/RaftKeeper/programs/server/Server.cpp:51:9 (raftkeeper+0x7e1cee)
2023-10-10T14:29:33.4670409Z     #20 RK::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&) /home/runner/work/RaftKeeper/RaftKeeper/programs/server/Server.cpp:161:5 (raftkeeper+0x7e3059) (BuildId: 43558d85c161f514)
2023-10-10T14:29:33.4671433Z     #21 Poco::Util::Application::run() /home/runner/work/RaftKeeper/RaftKeeper/contrib/poco/Util/src/Application.cpp:358:8 (raftkeeper+0xb3cf54) (BuildId: 43558d85c161f514)
2023-10-10T14:29:33.4672190Z     #22 RK::Server::run() /home/runner/work/RaftKeeper/RaftKeeper/programs/server/Server.cpp:94:25 (raftkeeper+0x7e25d5) (BuildId: 43558d85c161f514)
2023-10-10T14:29:33.4673003Z     #23 Poco::Util::ServerApplication::run(int, char**) /home/runner/work/RaftKeeper/RaftKeeper/contrib/poco/Util/src/ServerApplication.cpp:585:9 (raftkeeper+0xb61ee0) (BuildId: 43558d85c161f514)
2023-10-10T14:29:33.4673843Z     #24 mainEntryRaftKeeperServer(int, char**) /home/runner/work/RaftKeeper/RaftKeeper/programs/server/Server.cpp:264:23 (raftkeeper+0x7e4993) (BuildId: 43558d85c161f514)
2023-10-10T14:29:33.4674573Z     #25 main /home/runner/work/RaftKeeper/RaftKeeper/programs/main.cpp:98:12 (raftkeeper+0x7e10ca) (BuildId: 43558d85c161f514)
2023-10-10T14:29:33.4674917Z 
2023-10-10T14:29:33.4675722Z SUMMARY: ThreadSanitizer: heap-use-after-free /home/runner/work/RaftKeeper/RaftKeeper/contrib/poco/Net/include/Poco/Net/Socket.h:780:9 in Poco::Net::Socket::impl() const

How to reproduce

Run test_restore_from_snapshot and note that you should make node3 as initial leader. Also please note that the data race issue may not reproduce every time.

Update gtest version

If this doesn’t look right, choose a different type.

Description

gtest version is to old, update to 1.14.0.

CMake Deprecation Warning at contrib/googletest/googletest/CMakeLists.txt:56 (cmake_minimum_required):
  Compatibility with CMake < 2.8.12 will be removed from a future version of
  CMake.

Pulsar initialize-cluster-metadata have Operation 106 is unknown

Hello,I test a pulsar cluster use RaftKeeper.i deploy a RaftKeeper cluser with 3 node，I just modify configure id & host under
bin/pulsar initialize-cluster-metadata --cluster pulsar-test --metadata-store zk:ip1:8101,ip2:8101,ip3:8101 -configuration-metadata-store zk:ip1:8101,ip2:8101,ip3:8101 --web-service-url xxx --web-service-url-tls xx --broker-service-url xxx --broker-service-url-tls xxx
i have a error in RaftKeeper log：

2023.04.03 18:46:25.024026 [ 20125 ] {} ConnectionHandler: Error processing session 0x1 request.: Code: 999, e.displayText() = Coordination::Exception: Operation 106 is unknown (Unimplemented), Stack trace (when copying this message, always include the lines below):

/data1/home/wujianchao/project/jd/raftkeeper/contrib/libcxx/include/exception:133: Poco::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > const&, int) @ 0x9fdd8c in /usr/local/RaftKeeper/lib/raftkeeper
/data1/home/wujianchao/project/jd/raftkeeper/src/Common/Exception.cpp:56: RK::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > const&, int, bool) @ 0x6e9cd1 in /usr/local/RaftKeeper/lib/raftkeeper
/data1/home/wujianchao/project/jd/raftkeeper/src/Common/ZooKeeper/IKeeper.cpp:26: Coordination::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > const&, Coordination::Error, int) @ 0x8e8db6 in /usr/local/RaftKeeper/lib/raftkeeper
/data1/home/wujianchao/project/jd/raftkeeper/contrib/libcxx/include/string:1444: Coordination::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > const&, Coordination::Error) @ 0x8e8fc0 in /usr/local/RaftKeeper/lib/raftkeeper
/data1/home/wujianchao/project/jd/raftkeeper/src/Common/ZooKeeper/ZooKeeperConstants.cpp:0: Coordination::getOpNum(int) @ 0x8fa42e in /usr/local/RaftKeeper/lib/raftkeeper
/data1/home/wujianchao/project/jd/raftkeeper/src/Common/ZooKeeper/ZooKeeperIO.cpp:126: Coordination::read(Coordination::OpNum&, RK::ReadBuffer&) @ 0x8fac1c in /usr/local/RaftKeeper/lib/raftkeeper
/data1/home/wujianchao/project/jd/raftkeeper/src/Service/ConnectionHandler.cpp:615: RK::ConnectionHandler::receiveRequest(int) @ 0x75d876 in /usr/local/RaftKeeper/lib/raftkeeper
/data1/home/wujianchao/project/jd/raftkeeper/src/Service/ConnectionHandler.cpp:0: RK::ConnectionHandler::onSocketReadable(Poco::AutoPtrRK::ReadableNotification const&) @ 0x758269 in /usr/local/RaftKeeper/lib/raftkeeper
/data1/home/wujianchao/project/jd/raftkeeper/contrib/poco/Foundation/include/Poco/AutoPtr.h:96: Poco::NObserver<RK::ConnectionHandler, RK::ReadableNotification>::notify(Poco::Notification*) const @ 0x761f49 in /usr/local/RaftKeeper/lib/raftkeeper
/data1/home/wujianchao/project/jd/raftkeeper/contrib/libcxx/include/iterator:1496: Poco::NotificationCenter::postNotification(Poco::AutoPtrPoco::Notification) @ 0xa2b786 in /usr/local/RaftKeeper/lib/raftkeeper
/data1/home/wujianchao/project/jd/raftkeeper/contrib/poco/Foundation/include/Poco/AutoPtr.h:96: RK::SocketNotifier::dispatch(RK::SocketNotification*) @ 0x81a9b7 in /usr/local/RaftKeeper/lib/raftkeeper
/data1/home/wujianchao/project/jd/raftkeeper/src/Service/SocketReactor.cpp:292: RK::SocketReactor::dispatch(Poco::AutoPtrRK::SocketNotifier&, RK::SocketNotification*) @ 0x818355 in /usr/local/RaftKeeper/lib/raftkeeper
/data1/home/wujianchao/project/jd/raftkeeper/contrib/libcxx/include/atomic:1072: RK::SocketReactor::run() @ 0x816ca0 in /usr/local/RaftKeeper/lib/raftkeeper
/data1/home/wujianchao/project/jd/raftkeeper/contrib/poco/Foundation/src/Thread_POSIX.cpp:373: Poco::ThreadImpl::runnableEntry(void*) @ 0xa5cd55 in /usr/local/RaftKeeper/lib/raftkeeper
start_thread @ 0x7ea5 in /usr/lib64/libpthread-2.17.so
__clone @ 0xfeb0d in /usr/lib64/libc-2.17.so
(version v2.0.1)

Do you provide deployment guidance and images in the K8S environment?

If this doesn’t look right, choose a different type.
Do you provide deployment guidance and images in the K8S environment?

Describe your ideas

Do you provide deployment guidance and images in the K8S environment?

Solution

Allow document start for yaml

Description

Resources are separated by --- when there are multiple resources in one yaml file, which is needed by PR #52. If we set document-start to false, yamllint won't allow the existence of --- in yaml file. Can we set document-start to true to allow PR #52 passing style check?

Are you willing to submit PR?

yes.

[ Easy Tasks - Good for beginners ]

We appreciate any contributions from developers who are passionate about our project. We have some easy tasks that you can work on to get started. These tasks are a great way to learn more about our project and contribute to the community.

If you are interested in the tasks, please leave a comment below or on the issue page. We would love to hear from you and answer any questions you may have. Your contributions are greatly appreciated and will help us make our project a success. Thank you for your interest!

Refact project structure

Description
Now project structure is a little, we can make it better

Are you willing to submit PR?

Yes I am willing to submit a PR!

Code of Conduct
The Code of Conduct helps create a safe space for everyone. We require that everyone agrees to it.

I agree to follow this project's Code of Conduct *

Thanks for contributing to RaftKeeper!

jdraftkeeper / raftkeeper Goto Github PK

raftkeeper's Introduction

How to start?

1. Build RaftKeeper

Build on Ubuntu

2. Deploy RaftKeeper

3. Access RaftKeeper

How to migrate from Zookeeper?

raftkeeper's People

Contributors

Stargazers

Watchers

Forkers

raftkeeper's Issues

Description

How to achieve

Build on Ubuntu failed，Here are my steps，What should I do to build successfully

environmental information

step1

step2

step3

step4

step5

Step 5 executes an error report, modifies the CMakeLists.txt in the root directory of RaftKeeper, and adds the following configuration at the file entry

Re-execute after modification

Description

Are you willing to submit PR?

RaftKeeper version

Description

How to reproduce

Description

RaftKeeper version

Description

How to reproduce

Description

Are you willing to submit PR?

RaftKeeper version

Description

How to reproduce

Description

Are you willing to submit PR?

RaftKeeper version

Description

Describe your ideas

Solution

RaftKeeper version

How to reproduce

Analysis

How to fix？

Describe your ideas

Describe your doubt

RaftKeeper version

How to reproduce

Expected behavior

1 Issue: RaftKeeper Bug report

Search before asking *

Version *

What's the problem? *

What You Expected? *

How to Reproduce? *

Anything Else

Are you willing to submit PR?

Code of Conduct

2 Issue: RaftKeeper enhancement

Search before asking

Description

Solution

Are you willing to submit PR?

Code of Conduct

3 Issue: RaftKeeper feature request

Search before asking

Description

Use Case

Related Issue

Are you willing to submit PR?

Code of Conduct

Description

Are you willing to submit PR?

RaftKeeper version

Description