Comments (6)
Hi @Steamgjk
As you can find in the benchmark program, client and leader are running in the same process (more precisely, there is no separate client but the benchmark program itself is both client and server, and directly invokes the Raft API), to measure the pure Raft performance:
NuRaft/tests/bench/raft_bench.cxx
Lines 303 to 304 in f004f4c
Hence, there is no network cost between client and leader, and each replication can be done within a single RTT.
from nuraft.
Hi, @greensky00
I am wondering whether you have some bench results with >3 replicas, because according to my perf test,
with 3 replicas, the throughput can reach ~33K ops/second
but when it comes to 9 replicas, the throughput drops to ~20K ops/second
I am wondering whether this is a normal result, or I miconfigured something.
from nuraft.
Hi @Steamgjk
We don't publish the result with more replicas, but that is expected behavior as the amount of data that the leader should transfer over the network is proportional to the number of followers.
from nuraft.
Hi, @greensky00 . Do you think it is bounded by network bandwidth or something else? You know, I am using n1-standard-32 VM as replias, the bandwidth is 32^4 Gbps. I feel that is quite large and should be inefficient. So I think it may be more reasonable to attribute the bottleneck to CPU, because replicas need to serialize/deserialize/process more messages when we have more replicas. What do you think is a convincing explanation, CPU or bandwidht, or something else?
https://cloud.google.com/compute/docs/machine-types
from nuraft.
@Steamgjk
You can monitor the CPU usage of the leader during the test, and if it reaches 3200%, you can say it is CPU-bound. But I don't believe it uses that much CPU. Network bandwidth is also not likely the cause, as payload size (256 bytes) is small compared to the TCP window size (64KB by default). With simple math, the total throughput of the leader is 256 * 20K * 8 ~= 40MB only for 9 replicas.
Most likely the bottleneck comes from serialization. As I already mentioned in the other comment (#207 (comment)), it is not a random and independent data broadcasting. Replication should be strictly and globally ordered, which means log X
can't be sent to a follower without sending logs up to X-1
. Since there are more packets for "sending logs up to X-1
" with more followers, those increased packets are likely waiting for more time in the network queue, and consequently, "sending log X
" should wait for more time as well.
from nuraft.
Most likely the bottleneck comes from serialization. As I already mentioned in the other comment (#207 (comment)), it is not a random and independent data broadcasting. Replication should be strictly and globally ordered, which means log
X
can't be sent to a follower without sending logs up toX-1
. Since there are more packets for "sending logs up toX-1
" with more followers, those increased packets are likely waiting for more time in the network queue, and consequently, "sending logX
" should wait for more time as well.
I agree with it. Serialization/deserialization should be the bottleneck
from nuraft.
Related Issues (20)
- Clean up CMakeLists HOT 2
- nuraft 2.0.0 cannot work on centos7 + centos8 HOT 2
- Preconditions of apply_pack HOT 1
- handle_append_entries_resp() declined append HOT 1
- How to use NuRaft with CMake's FetchContent? HOT 12
- CMake targets should be namespaced using ALIAS targets
- CMake targets are missing usage requirements
- Leadership yielding is not synchronized with replicated log HOT 1
- Should the type be uint8_t instead of size_t for serialize_v1p(...) in srv_state.hxx line 133 HOT 1
- This is a question, not an issue. About `state_machine::pre_commit` HOT 2
- Does nuraft support linearizable read if generating no raft log entry of read requests? HOT 2
- Out of order call to state_machine::create_snapshot() when manually triggering a snapshot HOT 4
- Found a MSan error in Asio HOT 4
- Data race in peer HOT 3
- auto_fwd_resp_handler should forward resp->get_result_code
- src/tracer.hxx:52 stack-buffer-overflow when using vsnprintf's return value
- cmd_result::get_result_str missing `SERVER_IS_LEAVING`
- The semantics of the nuraft::snapshot HOT 3
- Merge two clusters into one HOT 1
- Misleading append_entries result code HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from nuraft.