test benchmark result question about nuraft

[greensky00]

Hi @zixin530

In that "Payload Size" test, we used a single client thread and put traffic at max speed, so that ops/s is always bounded by a single RTT latency, which is ~5K ops/s in our test environment as shown in the first test result.

When ops/s is fixed, the overall throughput increases according to each payload size (i.e., throughput = payload size * ops/s), but it will not increase forever as it will be bounded by either network bandwidth or NuRaft implementation itself (internally doing locking and memcpy). The purpose of this test is to see where the saturation point is.

[zixin530]

Thank you for your reply. Your explanation is very helpful. I do have one more question. For the latency test, I see 50% tile has a latency of 187us but when it comes to 99.99% it increases to 3746 us. If all of input request have the same payload size, why some requests are faster and others are way slower? Thank you.

[greensky00]

There can be a number of factors that influence tail latencies like p99.99 or p99.999: network latency itself, network queuing, VM environment (as there are many VMs in the same bare-metal and we don't know what other VMs are doing), or even thread sleep/wake-up may rarely take a few milliseconds.