When arbitrary command (simple set command) is used and run-count is set to be greater than 1, only the first run succeeds and subsequent runs (> 1) were found to be failing.
Redis server was replying with "-ERR unknown command" response.
Examining the TCP dump of the redis commands sent by memtier_benchmark suggests that the RESP was malformatted for runs > 1.

Test Setup
memtier_benchmark: master branch @f55b4b6
Redis: 5.0.7, standalone
Runs on the same host

To reproduce the issue

memtier_benchmark -s localhost -p 6379 --protocol=redis --clients=1 --run-count=3 --command="set __key__ __data__"

Log

[RUN #1] Preparing benchmark client...
[RUN #1] Launching threads now...
[RUN #1 74%,   1 secs]  4 threads:       29582 ops,   29573 (avg:   29573) ops/sec, 2.22MB/sec (avg: 2.22MB/sec),  0.13 (avg:  0.13) msec latency^M[RUN #1 100%,   1 secs]  0 threads:       40000 ops,   29573 (avg:   29831) ops/sec, 2.22MB/sec (avg: 2.24MB/sec),  0.13 (avg:  0.13) msec latency^M

[RUN #2] Preparing benchmark client...
[RUN #2] Launching threads now...
server 127.0.0.1:6379 handle error response: -ERR unknown command `*3  $3  set  `, with args beginning with: `memtier-6263819`, `xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx`,
server 127.0.0.1:6379 handle error response: -ERR unknown command `*3  $3  set  `, with args beginning with: `memtier-6263819`, `xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx`,
server 127.0.0.1:6379 handle error response: -ERR unknown command `*3  $3  set  `, with args beginning with: `memtier-6263819`, `xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx`,
... Repeated ...
[RUN #2 100%,   1 secs]  0 threads:       40000 ops,   24478 (avg:   23277) ops/sec, 4.80MB/sec (avg: 4.57MB/sec),  0.14 (avg:  0.15) msec latency^M

[RUN #3] Preparing benchmark client...
[RUN #3] Launching threads now...
server 127.0.0.1:6379 handle error response: -ERR unknown command `*3  $13  *3  $3  set    `, with args beginning with: `memtier-6263819`, `xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx`,
server 127.0.0.1:6379 handle error response: -ERR unknown command `*3  $13  *3  $3  set    `, with args beginning with: `memtier-6263819`, `xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx`,
server 127.0.0.1:6379 handle error response: -ERR unknown command `*3  $13  *3  $3  set    `, with args beginning with: `memtier-6263819`, `xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx`,
... Repeated ...
[RUN #3 100%,   1 secs]  0 threads:       40000 ops,   24587 (avg:   24110) ops/sec, 5.34MB/sec (avg: 5.24MB/sec),  0.14 (avg:  0.14) msec latency^M

RESP captured for run>1

=== Request ===

=== Response ===

I would add it to the list of pre-reqs

Hi,

I wanted to install memtier_benchmark tool but it returned to a torture.I followed the instructions in README file but lots of packages are not exists in Centos6.4.
I installed below packages

yum install autoconf (http://petio.org/tools/autoconf.html)
yum install automake (http://petio.org/tools/automake.html)
yum install libmemcached
yum install libmemcached
yum install libmemcached-devel
After retrying ./configure gave below result.

No package 'libevent' found

Consider adjusting the PKG_CONFIG_PATH environment variable if you
installed software in a non-standard prefix.

Alternatively, you may set the environment variables LIBEVENT_CFLAGS
and LIBEVENT_LIBS to avoid the need to call pkg-config.
See the pkg-config man page for more details.

Than I try to install libevent but I got the conflicts

Transaction Check Error:
file /usr/bin/event_rpcgen.py from install of libevent-devel-2.0.19-1.rhel6.x86_64 conflicts with file from package compat-libevent14-1.4.13-1.rhel6.x86_64
file /usr/lib64/libevent_core.a from install of libevent-devel-2.0.19-1.rhel6.x86_64 conflicts with file from package compat-libevent14-1.4.13-1.rhel6.x86_64
file /usr/lib64/libevent_core.so from install of libevent-devel-2.0.19-1.rhel6.x86_64 conflicts with file from package compat-libevent14-1.4.13-1.rhel6.x86_64
file /usr/lib64/libevent_extra.a from install of libevent-devel-2.0.19-1.rhel6.x86_64 conflicts with file from package compat-libevent14-1.4.13-1.rhel6.x86_64
file /usr/lib64/libevent_extra.so from install of libevent-devel-2.0.19-1.rhel6.x86_64 conflicts with file from package compat-libevent14-1.4.13-1.rhel6.x86_64

When I try to erase compat-libevent14 package it shows that it is a dependency of nfs-utils.
Removing:
compat-libevent14 x86_64 1.4.13-1.rhel6 @pgdg91 466 k
Removing for dependencies:
nfs-utils x86_64 1:1.2.3-36.el6 @base 977 k
nfs-utils-lib x86_64 1.1.5-6.el6 @base 149 k

So I just cancelled installation of memtier_benchmark.And if it is possible can you prepare a tutorial for a centos release?

Regards

0、network latency
The network latency is around 0.15ms。

1、redis-cli latency
wanghaiyang@db-ck-1c-1:~$ redis-cli -h redis-bmk.uyffb3.ng.0001.apne1.cache.amazonaws.com --latency

2、memtier_benchmark result

memtier_benchmark -s redis-bmk.uyffb3.ng.0001.apne1.cache.amazonaws.com -t 2 -c 5 --ratio=1:6 --test-time=120 --select-db=0 --hide-histogram -x 3 --expiry-range=60-90 --data-size=384

The latency of memtier_benchmark is less than redis-cli latency and network latency，why？

I locally benchmark a Redis server 5.0 with the following command:
memtier_benchmark --show-config -d 4096 --key-maximum=50000 --hide-histogram -t 8 --ratio=1:1 --key-pattern=P:P --run-count 10

So, I have 50K keys and values of size 4K. The maximum size of my Redis instance must be 50K*4KB ~= 200MB.

However, with each next run of memtier-benchmark, my Redis instance size increases by 200MB. After 10 runs, Redis eats 10*200MB = 2GB of RAM.

I assumed that Redis must replace the same keys and delete old values, so its size is always capped at ~200MB. Why do I see 2GB then? Is this the intended behavior of --run-count?

the test is like this.

run memtier on redis pack DB with replication with default endpoint bind.
kill endpoint and master redis node, failover of endpoint works well and slave is promoted well. memtier, even if memtier does multiple run ( using -x parameters ) it will never connect to the new endpoint.

restart memtier works and connect correctly. it will be great if memtier can refresh its connection to the new endpoint automatically.

Hi,

When I've tried to run memtier with --random-data flag, I've noticed that the generated data is not really random. So, I've looked at the code that allocates the value buffer -

1. Can you please explain the rational behind this formula?
2. Why didn't you generate the random data by simply reading the whole buffer from /dev/urandom?

Thanks,
Adir

The --test-time option forces memtier_benchmark to finish after test-time seconds.

However, I noticed that each client actually waits to receive a reply from a Redis server, then checks if test-time elapsed, and then stops. In my case, I have many parallel clients and many stragglers, where a single Redis reply can take several seconds to come back to client.

Thus, even though I specify --test-time=10, it takes significantly more than 10 seconds. This not only takes too long for one experiment run, but also skews the results. (Which may be the behavior the authors desired?..)

I believe that all clients must die immediately after test-time seconds. I fixed this issue with the following trivial patch:

diff --git a/client.cpp b/client.cpp
index e827195..f074877 100755
--- a/client.cpp
+++ b/client.cpp
@@ -557,7 +557,18 @@ int client_group::prepare(void)

 void client_group::run(void)
 {
+    // exit event loop immediately after test_time seconds
+    if (m_config->test_time > 0) {
+        struct timeval endtime;
+        endtime.tv_sec  = m_config->test_time;
+        endtime.tv_usec = 0;
+        event_base_loopexit(m_base, &endtime);
+    }
+
     event_base_dispatch(m_base);
+
+    for (std::vector<client*>::iterator i = m_clients.begin(); i != m_clients.end(); i++)
+        (*i)->set_end_time();
 }

 unsigned long int client_group::get_total_bytes(void)

https://hub.docker.com/r/redislabs/memtier_benchmark
Current master Dockerfile builds 1.2.15 and Hub's is 1.2.8.
Perhaps setting an Automated Build would be better. It would build always master branch to :latest image tag and would create individual image tags for each additional git tag.
See https://docs.docker.com/docker-hub/builds/

I'm trying to compile it on FreeBSD and get an error:

make all-am
CXX memtier_benchmark-memtier_benchmark.o
CXX memtier_benchmark-client.o
CXX memtier_benchmark-cluster_client.o
CXX memtier_benchmark-shard_connection.o
CXX memtier_benchmark-run_stats_types.o
In file included from run_stats_types.cpp:21:
In file included from ./run_stats_types.h:22:
In file included from ./memtier_benchmark.h:23:
./config_types.h:87:5: error: unknown type name 'pthread_mutex_t'; did you mean 'pthread_mutex'?
pthread_mutex_t m_mutex;
^~~~~~~~~~~~~~~
pthread_mutex
/usr/include/stdio.h:154:9: note: 'pthread_mutex' declared here
struct pthread_mutex _fl_mutex; / used for MT-safety */
^
In file included from run_stats_types.cpp:21:
In file included from ./run_stats_types.h:22:
In file included from ./memtier_benchmark.h:23:
./config_types.h:87:21: error: field has incomplete type 'pthread_mutex'
pthread_mutex_t m_mutex;
^
/usr/include/stdio.h:154:9: note: forward declaration of 'pthread_mutex'
struct pthread_mutex _fl_mutex; / used for MT-safety */
^
2 errors generated.
*** Error code 1

Stop.

I am trying to run memtier_benchmark on mac os. My Redis instances are running on my local machine which runs mac os

1. Trying with Docker
   Running tests for a single instance(non cluster-mode) works fine after using -net=host. But when using --cluster-mode flag I am getting connection refused error. I guessed it is because CLUSTER SLOTS command will return ip as 127.0.0.1 from my local machine and then memtier uses it inside docker image and fails. To avoid that I set up rerouting of IP 127.0.0.1 to my local machines IP. This seemed working, but Redis started giving error "Error: Connection reset by peer" when using redis-cli from my docker image.
   I gave up at this point

2. Building from source file
   ./configure command gives an error at with this log

checking for pcre_compile in -lpcre... yes
checking for deflateInit_ in -lz... yes
checking for clock_gettime in -lrt... no
configure: error: rt is required for libevent

Can you please point me in the right direction?

Hi memtier_benchmark developers,

Currently memtier_benchmark can report average latency in the results, which is usually enough for simple benchmarks. it also provides the option to print the detail latency distribution. The data is aggregated through different latency level. However for storage/DB performance, people will also care about the tail latency(99%, 99.9% or 99.99%, example from fio). Is it possible to provide these tail latency data also?

ALL STATS
========================================================================
Type        Ops/sec     Hits/sec   Misses/sec      Latency       KB/sec
------------------------------------------------------------------------
Sets       21536.09          ---          ---      0.23100     22561.95
Gets           0.00         0.00         0.00      0.00000         0.00
Waits          0.00          ---          ---      0.00000          ---
Totals     21536.09         0.00         0.00      0.23100     22561.95

thanks, -yuan

The memtier_benchmark tool is using the -s --server as a parameter to pass the server address.

It would be good to be consistent with other redis-* tools and use -h --hostname instead

When running the memtier_benchmark client with either a couple more threads or a couple more connections than the default, at some point I got an accept4(): No file descriptors available error and after some time the benchmark does not progress any more. Often this goes along with read error: Connection reset by peer messages printed in the beginning.

Here an example for my local system and a custom Docker container:

$ docker run --rm -ti quay.io/kinvolk/memtier:latest-amd64
/ # memcached -u root &
/ # memtier_benchmark -p 11211 -P memcache_binary -t 100 --test-time=3 -x 1
[RUN #1] Preparing benchmark client...
accept4(): No file descriptors available
[RUN #1] Launching threads now...
[RUN #1 20%,   0 secs] 81 threads:      877668 ops, 1456138 (avg: 1439975) ops/sec, 55.07MB/sec (avg: 54.31MB/sec),  3.42 (avg:  3.40) msec latencyy
^C
[Stopped it at this point with Ctrl-C]
/ # memtier_benchmark -p 11211 -P memcache_binary -t 4 -c 400 --test-time=3 -x 1
[RUN #1] Preparing benchmark client...
accept4(): No file descriptors available
[RUN #1] Launching threads now...
[RUN #1 63%,   1 secs]  4 threads:      712807 ops,       0 (avg:  379203) ops/sec, 0.00KB/sec (avg: 14.30MB/sec),  -nan (avg:  4.19) msec latencyy
^C [Similar problem here, stopped after some time]

root@simon-d14-1:~/benchmark/memtier_benchmark-1.2.7# ./configure
................
checking for gai_strerror in -lsocket... no
./configure: line 5344: syntax error near unexpected token LIBEVENT,' ./configure: line 5344:PKG_CHECK_MODULES(LIBEVENT,'

root@simon-d14-1:~/benchmark/memtier_benchmark-1.2.7# cat /etc/*release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=15.10

It would be great if the rate limit (throttling) was configurable

I generated the file in the same format by referring to yours. Below is an example of the first few entries.

dumpflags, time, exptime, nbytes, nsuffix, it_flags, clsid, nkey, key, data
0, 0, 0, 200, 200, 0, 1, 200, sqjbmtzgladigkgckqxhqvkvnvcdrctjfqvsfnkwcocgagwvfbsbpadxxirdnkfexzifhmqxriaqvwiwxegholwgheabfqdhvxyqquaoxqrcebhjlvvsurgofgaapmqgyvtngtfkhiowfezfsjsyspzdffaojbwwqombzlvwhqqxdschnjgcjtgfmdxmvpmujreflbvc, ddcihrvrhfykwaprhhoxrgqcgcekmxkfkptrkhsvptkgnxqrazainyabxdkznmsgrpsgyeuxzkdopkpioevdmifibezsdhvglfjfojctdrxwlosunongwfpwrsuklfxrkneovbbhkspbmpsqloczpjzoiumspoxhtrqttsyhbrqgbhmibpdzpkboxrpshnylickresgq
0, 0, 0, 200, 200, 0, 1, 200, voaidlhyacrsxwurfwbdharbcuiotsaabyjhohqrperazwfialssbekpdhjujsfosvqvdtkvdocghakxqfxpbjrlpacixygnforgvpkojbylonzynvnezmhenlkmmerisipveqfrefpmgmnumtwtckdvlklegtcmygibekvktjmikqvobepdrdahcvildjprbgtlowam, fujwzhpvjutlhxycaivguimukkkhtvvkwktbtuhthudapksgynusznwnbaarjjqtdroukejmcgfdjzcerfjbbaoclgattmokezeeooptggpqdwagcltwqhbdnhsyxstxfdsgoemsbnbjyjgtmnklklhvjspsxbcnaffsofxrlxzgjwppdjyaioonzzxbhrsydivfmlkf

Note : The strings are not separated by newline they are just separated by the comma. There is a newline only after one complete entry

I want to insert these as key value pairs using memcache_text.
I am using the following command:
memtier_benchmark -p 11211 -P memcache_text -t 4 -n 100 –ratio 1000:0 -c 25 -x 5 --data-import=key.csv

Note: file name is key.csv

But I am getting this error-
key.csv:2: error parsing item values.

How to insert these key-value pairs using memtier_benchmark?

I am using memtier_benchmark to verify a private memcashed.
I would like to verify performance with diffrent loads, such as (70% get/30% read).
ideally with random keys and random values.
However I would like to ensure that hit ratio is 100%, i.e all gets are for keys already set.
How/If can i control this in memtier_benchmark?

steps to reproduce

memtier_benchmark -n 10 --cluster-mode -p 20000 --run-count 3

segfault backtrace

Thread 6 "memtier_benchma" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fffeffff700 (LWP 27658)]
0x000055555558a9bf in mbulk_size_el::get_next_mbulk (this=0x7fffe8001a50) at protocol.h:75
75	            if (next->bulks_count == 0) {
(gdb) info registers
rax            0x30	48
rbx            0x7fffe8001b10	140737085709072
rcx            0x3	3
rdx            0x0	0
rsi            0x7fffe80008d0	140737085704400
rdi            0x7fffe8001a50	140737085708880
rbp            0x7fffefffec20	0x7fffefffec20
rsp            0x7fffefffec20	0x7fffefffec20
r8             0x7fffe80008d0	140737085704400
r9             0x0	0
r10            0x7fffe80008d0	140737085704400
r11            0x0	0
r12            0xe	14
r13            0x555555adbbc0	93824998030272
r14            0x7ffff7995d40	140737347411264
r15            0xe	14
rip            0x55555558a9bf	0x55555558a9bf <mbulk_size_el::get_next_mbulk()+27>
eflags         0x10206	[ PF IF RF ]
cs             0x33	51
ss             0x2b	43
ds             0x0	0
es             0x0	0
fs             0x0	0
gs             0x0	0
(gdb) bt
#0  0x000055555558a9bf in mbulk_size_el::get_next_mbulk (this=0x7fffe8001a50) at protocol.h:75
#1  0x0000555555588934 in redis_protocol::parse_response (this=0x55555654be20) at protocol.cpp:561
#2  0x0000555555571be8 in shard_connection::process_response (this=0x55555654bd90) at shard_connection.cpp:379
#3  0x0000555555570b84 in cluster_client_read_handler (bev=0x555555adb9a0, ctx=0x55555654bd90) at shard_connection.cpp:64
#4  0x00007ffff7995e32 in ?? () from /usr/lib/x86_64-linux-gnu/libevent-2.1.so.6
#5  0x00007ffff799b8f8 in ?? () from /usr/lib/x86_64-linux-gnu/libevent-2.1.so.6
#6  0x00007ffff799c33f in event_base_loop () from /usr/lib/x86_64-linux-gnu/libevent-2.1.so.6
#7  0x0000555555569dd7 in client_group::run (this=0x5555557c7140) at client.cpp:587
#8  0x000055555555cc08 in cg_thread_start (t=0x555555e888b0) at memtier_benchmark.cpp:996
#9  0x00007ffff700d6db in start_thread (arg=0x7fffeffff700) at pthread_create.c:463
#10 0x00007ffff63a088f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

When running ./configure we get:

...
checking for clock_gettime in -lrt... no
configure: error: rt is required for libevent.

I am trying to work with Memcached, but I get "error: response parsing failed".

protocol.cpp:477: unsupported response: 'ERROR'.
error: response parsing failed.

hi there, running memtier with the Gaussian distribution option on macOS is stalling it ( build with TLS support is disabled ).

memtier_benchmark --hide-histogram  --key-pattern=G:G

Output:

$ memtier_benchmark --hide-histogram  --key-pattern=G:G 
[RUN #1] Preparing benchmark client...
[RUN #1] Launching threads now...
[RUN #1 0%,   0 secs]  4 threads:           0 ops,       0 (avg:       0) ops/sec, 0.00KB/sec (avg: 0.00KB/sec),   nan (avg:   nan) msec latency

I'd like to verify the output data of a Memcached server (using --data-verify) using binary protocol. The --data-verify flag requires to import a file using --data-import.
I've tried creating a file following the structure described in README.import, but I keep getting an error.
Would someone be able to provide a sample file for --data-import, and what the expected output of memtier_benchmark would be when running --data-verify with the file?

Hi,
I am testing my middleware with for Multi-GETs in memcache-text protocol
Acc. to memtier_benchmark --help ,
--multi-key-get=NUM Enable multi-key get commands, up to NUM keys (default: 0)

Let's say multi-key-get is 5 in my case.
While running the experiments, I always see Multi-GETs with exactly 5 keys.
While this is technically correct, can we always assume that we will get exactly NUM keys in the
multi-get or this is not the expected behaviour?

Thanks

Hello,
Please check result of execution (in attachment).

1. In config specified only one client: ""clients": 1"
2. It means that requests sequentially sending in one thread: Request 1 sent, Response 1 got, Request 2 sent, Response 2 got...

Memtier statistics:
,"Gets":{
"Ops/sec": 50.20
,"Hits/sec": 43.70
,"Misses/sec": 6.50
,"Latency": 39.83
,"KB/sec": 46.10
}

There is only one thread and 50.2 requests per 1 second. 1 second it is 1000ms. Let's calculate how much time in ms spended on each request: 1000/50.2 = 19.92. I
1.zip

.e. in memtier 1 request takes 19.92ms (it time incudes: request generating, logging and all internal memtier's logic).
I think that latency should be less than 19.2, because it should be the time only for request, but we have Latency=39.83.
I think that, Latency * "Ops/sec" should be less than 1000. But we have 39.83*50.2=1999.466

Regards,
Minas

When I invoke memtier_benchmark with the --unix-socket option, it segfaults when the client is trying to connect to Redis. For some reason, it tries to invoke get_connection_info on the server_addr, but it is null. Here is the full backtrace:

#0  0x00007ffff7dc4365 in pthread_mutex_lock () from /lib/ld-musl-x86_64.so.1
#1  0x000055555556b716 in server_addr::get_connect_info (this=0x0, 
    ci=ci@entry=0x7fffffffd190) at config_types.cpp:234
#2  0x000055555555ca6d in client::connect (this=0x7ffff7b27d60)
    at client.cpp:199
#3  0x000055555555ca01 in client::prepare (this=<optimized out>)
    at client.cpp:344
#4  0x000055555555d52e in client_group::prepare (this=0x7ffff7d71fa0)
    at client.cpp:539
#5  0x000055555555b259 in cg_thread::prepare (this=0x7ffff7d71ea0, 
    this=0x7ffff7d71ea0) at memtier_benchmark.cpp:790
#6  run_benchmark (run_id=1, cfg=0x7fffffffe4f0, obj_gen=0x555555788c40)
    at memtier_benchmark.cpp:842
#7  0x0000555555559c5c in main (argc=<optimized out>, argv=<optimized out>)
    at memtier_benchmark.cpp:1160

Hi,

In run_benchmark() (memtier_benchmark.cpp), duration is calculated as following:
float factor = ((float)(thread_counter - 1) / thread_counter);
duration = factor * duration + (float)(*i)->m_cg->get_duration_usec() / thread_counter ;

I would like to understand which theory the above formula is based on in order to obtain the run duration of multiple threads?

Any insight will be very much appreciated!

Thanks.

how to run it in multiple docker container cluster

$ for p in seq 1 10 ;do /apps/svr/redis/bin/redis-cli -h 10.201.111.111 -p 6379 -a c4uT62vgCajbnvtgYFbq flushall ;./memtier_benchmark -s 10.201.111.111 -p 6379 -a c4xxxxxxxxxbq -n 10000 -c 16 -t 32 --reconnect-interval=10 | head -n 15 ;done
a lot of 'connect: connection failed: Connection timed out' error information in benchmark outputs.

-d --data-size=SIZE Object data size (default: 32)

What are the units for data size? Bytes?

From looking at memtier_benchmark's source code it didn't appear to support TLS/SSL connections to Redis. Are there any plans to implement this?

The updated new version does not seem to have the Makefile for running make command that is present in the README.
$ make
make: *** No targets specified and no makefile found. Stop

I can't run memtier_benchmark using Docker

docker run --rm memtier_benchmark -s localhost -p 11222 -network networkname  --requests 4

(error: requests must be greater than zero.)
Not sure what I'm doing wrong...

In my test enviroment, I got the following outputs:
$ ./memtier_benchmark -s 192.168.111.126 -p 2001 -n 1 -c 1 -t 2 -d 20000 --cluster-mode
[RUN #1] Preparing benchmark client...
[RUN #1] Launching threads now...
[RUN #1 100%, 0 secs] 2 threads: 2 ops, 6557 (avg: 6557) ops/sec, 125.38MB/sec (avg: 125.38MB/sec), 0.39 (avg: 0.39) msec latency

It will hang here like forever. By reading the code, it seems it should exit and output like this:
[RUN #1] Preparing benchmark client...
[RUN #1] Launching threads now...
[RUN #1 100%, 0 secs] 0 threads: 2 ops, 0 (avg: 6756) ops/sec, 0.00KB/sec (avg: 129.20MB/sec), 0.00 (avg: 0.37) msec latency

2 Threads
1 Connections per thread
1 Requests per client

ALL STATS

Type Ops/sec Hits/sec Misses/sec MOVED/sec ASK/sec Latency KB/sec

Sets 6779.66 --- --- 0.00 0.00 0.37200 132746.29
Gets 0.00 0.00 0.00 0.00 0.00 0.00000 0.00
Waits 0.00 --- --- --- --- 0.00000 ---
Totals 6779.66 0.00 0.00 0.00 0.00 0.37200 132746.29

Request Latency Distribution
Type <= msec Percent

SET 0.350 50.00
SET 0.390 100.00

Why? My test environment is cluster mode.

I am running memcached in MAC and using docker to benchmark memcached running on my local MAC. it's using latest in the master branch

here is the command:
docker run --rm memtier_benchmark -P memcache_text -p 11211 -s host.docker.internal --key-pattern=S:S --ratio=0:1

cmd put is

➜  memtier_benchmark git:(master) docker run  --rm memtier_benchmark -P memcache_text -p 11211 -s host.docker.internal --key-pattern=S:S --ratio=0:1
[RUN #1] Preparing benchmark client...
[RUN #1] Launching threads now...
[RUN #1 100%, 165 secs]  0 threads:     2000000 ops,   13225 (avg:   12067) ops/sec, 1.03MB/sec (avg: 963.70KB/sec), 15.14 (avg: 2164.05) msec latencyyycyy

4         Threads
50        Connections per thread
10000     Requests per client


ALL STATS
=========================================================================
Type         Ops/sec     Hits/sec   Misses/sec      Latency       KB/sec
-------------------------------------------------------------------------
Sets            0.00          ---          ---      0.00000         0.00
Gets        12066.21     12066.21         0.00   2164.04600       963.62
Waits           0.00          ---          ---      0.00000          ---
Totals      12066.21     12066.21         0.00   2164.04600       963.62

some value for histogram output

GET      50.000        99.93
GET      51.000        99.94
GET      52.000        99.94
GET      53.000        99.94
GET      54.000        99.94
GET      55.000        99.95
GET      56.000        99.95
GET      57.000        99.95
GET      58.000        99.95
GET      59.000        99.95
GET      60.000        99.95
GET    4300000.000       100.00

However, I am always getting 4300000.000 at the very end for GET. (same for SET).
which is probably causing my avg latency pretty high.

Can someone explain why I am always getting 4300000

If multiple threads and random write options are provided, all the threads perform the operation to same set of keys.
This is because the random_generator initializes the state of the random number generator by providing the same seed. So, all the threads initialize the generator with the same seed thus end up creating same set of keys.

int ret = initstate_r(1, m_state_array, sizeof(m_state_array), &m_data_blob);

I think the seed should be separate across threads.

Hi,

Thanks for developing and maintaining the great tool!

I was trying this tool with memcached and encountered a lot of "Connection error: Connection timed out". These errors weren't quite surprising to me since my memcached server was instrumented by an intel pintool and thus expected to be very slow when handling connections. So I was wondering if there's any way I can increase the timeout value for the benchmark tool to connect to the server. I figured out there's a libmemcached option "MEMCACHED_BEHAVIOR_CONNECT_TIMEOUT" that seems relevant, but I have no idea how to use it in the benchmark tool.

Any help/thought would be appreciated!

While performing benchmarking on time bound scenarios with heavy data sizes in the range of 8MBs or higher and with higher pipelines of >=8, I am noticing assertion failure.

Example command:
./memtier_benchmark -s localhost -p 6379 --test-time 300 -d 8000000 --pipeline=8 --threads=1 -c 20 --key-pattern=S:S --ratio=1:1
Error:
memtier_benchmark: client.cpp:426: void client::handle_event(short int): Assertion `finished() == false' failed.), -nan (avg: 20648.63) msec

This does not happen when time constraint is removed with memtier ( even though I select higher data sizes and pipelines ).
Also visiting the source code reveals, while finishing the tests, its trying to see if the write-buffer has any bytes left and if it is, then this assertion is thrown. Is this an intentional logic to show that benchmarking did not run clean when the time was fixed.

System details:
Intel DL 380 (Ubuntu 16.04 with default kernel)
RAM: 256GB , free ~247GB

The throughput number(5272.41 KB/sec) in the final table is incorrect.

$./memtier_benchmark --protocol redis -x 1 -s 209.243.160.17 -p 6379 --ratio=1:0 -t16 -c4 --key-pattern=R:R --key-minimum=1 --key-maximum=100000000 -n 15000 -d 32768[RUN #1] Preparing benchmark client...
[RUN #1] Launching threads now...
[RUN #1 100%, 266 secs] 0 threads: 960000 ops, 3549 (avg: 3604) ops/sec, 111.09MB/sec (avg: 112.81MB/sec), 18.08 (avg: 17.73) msec latency

16 Threads
4 Connections per thread
15000 Requests per thread
Type Ops/sec Hits/sec Misses/sec Latency KB/sec
Sets 3595.90 --- --- 17.72600 5272.41
Gets 0.00 0.00 0.00 0.00000 0.00
Totals 3595.90 0.00 0.00 17.72600 5272.41

Hi,
I am getting 0 hits/sec if i am running this benchmark on newly installed redis and memcache. Can you explain why this is so and how can i improve the hit percentage. Can you tell for what configurations what will be the result. Please help.

Thank you,
Kirti.

Similar to how redis-benchmark works - would be extremely useful for module authors...

What did I do wrong？

It does not occur when not using cluster_mode or when connecting to a local server.

It happened in the following environment.

bootjp@server:~$ memtier_benchmark --version
memtier_benchmark 1.2.17
Copyright (C) 2011-2017 Redis Labs Ltd.
This is free software.  You may redistribute copies of it under the terms of
the GNU General Public License <http://www.gnu.org/licenses/gpl.html>.
There is NO WARRANTY, to the extent permitted by law.

bootjp@server:~$ uname -a
Linux server 4.15.0-72-generic #81-Ubuntu SMP Tue Nov 26 12:20:02 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
bootjp@server:~$ cat /etc/os-release
NAME="Ubuntu"
VERSION="18.04.3 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04.3 LTS"
VERSION_ID="18.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic
bootjp@server:~$

bootjp@server:~$ lscpu
Architecture:        x86_64
CPU op-mode(s):      32-bit, 64-bit
Byte Order:          Little Endian
CPU(s):              24
On-line CPU(s) list: 0-23
Thread(s) per core:  2
Core(s) per socket:  6
Socket(s):           2
NUMA node(s):        2
Vendor ID:           GenuineIntel
CPU family:          6
Model:               45
Model name:          Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz
Stepping:            7
CPU MHz:             1676.846
CPU max MHz:         2500.0000
CPU min MHz:         1200.0000
BogoMIPS:            4000.00
Virtualization:      VT-x
L1d cache:           32K
L1i cache:           32K
L2 cache:            256K
L3 cache:            15360K
NUMA node0 CPU(s):   0-5,12-17
NUMA node1 CPU(s):   6-11,18-23
Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx lahf_lm epb pti ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid xsaveopt dtherm ida arat pln pts md_clear flush_l1d

Is there any other information I need?

I am trying to execute third party rest api from memtier it give below error while making the build
protocol.cpp:40:9: error: ‘SimpleWeb’ does not name a type
typedef SimpleWeb::ClientSimpleWeb::HTTP HttpClient;

So I am trying to use client_http.hpp to call rest api . But I am not able to create its instance.
I have include header file in protocol.cpp and in makefile

We've run into an interesting deadlock problem when implementing a middleware between memtier_benchmark and memched for a university project. We are three people with three independent implementations and we all have the same observation (see bold part below), so I'd like to ask you a few questions in order to gain more insights into memtier_benchmark and the deadlock problem.

We are to implement our middleware using a special architecture: a thread should accept connections and put these into a Java BlockingQueue that is then processed by a number of worker threads who forward the request to the memcached server and finally reply to memtier_benchmark. There is a simple illustration for this on page 6 on the Project description. We all decided to serve a request from memtier_benchmark and then put the (still open) connection back into the queue.

The following works with any number of worker threads in the middleware. Here I will consider the case with a single worker thread serving memtier_benchmark requests. I use 1 memtier thread having 200 clients and releasing 10 requests of data size 16.

Starting memtier_benchmark. It works fine until it reaches about 95% progress, then everything blocks. Analysis showed that all three implementations block on reading on a connection from memtier_benchmark. For some reason, memtier_benchmark suddenly doesn't send any data on a given connection any more. This is the phenomenon we want to understand.

We have a possible explanation, but it's highly implementation specific for memtier_benchmark. This is why I'm writing this GitHub issue: we hope to get some light into this mystery. Our tentative explanation is as follows:

• Without loss of generality, let's assume has that memtier_benchmark has a single thread sending out requests and that all open connections ("virtual clients") are stored in a queue. Consider 10 connections between memtier_benchmark and our middleware. Both sides (according to our hypothesis) buffer the connections in a queue.
• At time x (after an arbitrary number of requests have been exchanged and both queues are filled with open connections), the following situation appears:
  • Our middleware replies to a request in connection 3, puts it back into the queue and takes from the queue, getting connection 4. It contacts memcached, gets a reply and forwards the reply to the connection 4. It then puts connection 4 back into the queue and takes connection 5 from the queue.
  • memtier_benchmark takes connection 3 from the queue, checks our reply and sends a new request on connection 3. It then puts the connection 3 back into the queue and takes connection 2 out of the queue. It sends a request and waits for the reply.
  • This means that memtier_benchmark is waiting on connection 2 and our middleware is waiting on connection 5, which would effectively result in a benchmark.

We make bold assumptions about memtier_benchmark's internals in this example. Would such a scenario be possible with memtier_benchmark? Do you have an idea why the read on a connected memtier_benchmark blocks? Thank you very much in advance, we are thrilled by the problem and looking forward to hear your input.

Steps to reproduce:

1. Take any redis release ( tested on latest redis-4.0.6 as well) and install it on Linux OS ( Redhat/Ubuntu)
2. Start the redis server process
3. Take any memtier release version 1.2.8 or onwards and install it.

Tested platforms : x86 , QDF2400

Example command to run:
memtier_benchmark -s localhost -p 6379 -d 2 --pipeline=10 --threads=10 -c 50 --key-pattern=S:S --ratio=1:1 -n 10000 --out-file ./Memtier_out

PS : key-pattern=S:S ( i.e. sequential operations for read or write )

Hits/sec will always be 0 ( attached snapshot of this run MemtierIssue.jpg

)

This happens because as per the code change that has gone into 1.2.8+ affecting the client::create_request(struct timeval timestamp)
where the iter variable is not correctly updated for S operation get methods.

Introducing function : obj_iter_type(benchmark_config cfg, unsigned char index) in version 1.2.8+ to reduce redundant code for setting OBJECT_GENERATOR_ variable at everyplace,
For S operations, it always returns OBJECT_GENERATOR_KEY_SET_ITER and there is a NO assignment of iter = OBJECT_GENERATOR_KEY_GET_ITER for get method.

Hence, there is always a miss and hits/sec=0.

This happens for any key-pattern like S:S except R,G operations.

Attaching the patch for fixing this issue.
client.zip

Thanks,
Ashutosh Parida
Qualcomm Datacenter Technologies

the latency and the throughput are not matching.
the machines are free without any application process.

1. test with no password. has a low latency and less throughput

$ for p in seq 1 10 ;do /apps/svr/redis/bin/redis-cli -h 10.201.111.111 -p 6379 -a c4uT62vgCajbnvtgYFbq flushall ;./memtier_benchmark -s 10.201.111.111 -p 6379 -a c4uT62vgCajbnvtgYFbq -n 5000 -c 32 -t 4 --reconnect-interval=5 | head -n 10 ;done
OK
[RUN #1] Preparing benchmark client...
[RUN #1] Launching threads now...
[RUN #1, 10 secs] 0 threads: 640000 ops, 62483 ops/sec, 2.32MB/sec, 0.70msec latency
4 Threads
32 Connections per thread
5000 Requests per thread
Type Ops/sec Hits/sec Misses/sec Latency KB/sec
Sets 5689.69 --- --- 0.70000 438.33
Gets 56834.37 0.00 56834.37 0.69700 1936.84
Totals 62524.06 0.00 56834.37 0.69700 2375.17

2. test with password has a high latency and more throughput

$ for p in seq 1 10 ;do /apps/svr/redis/bin/redis-cli -h 10.201.111.111 -p 6379 flushall ;./memtier_benchmark -s 10.201.111.111 -p 6379 -n 5000 -c 32 -t 4 --reconnect-interval=5 | head -n 10 ;done
OK
[RUN #1] Preparing benchmark client...
[RUN #1] Launching threads now...
[RUN #1, 9 secs] 0 threads: 640000 ops, 69699 ops/sec, 2.59MB/sec, 1.78msec latency
4 Threads
32 Connections per thread
5000 Requests per thread
Type Ops/sec Hits/sec Misses/sec Latency KB/sec
Sets 6410.54 --- --- 1.77800 493.86
Gets 64034.96 0.00 64034.96 1.77500 2182.23
Totals 70445.50 0.00 64034.96 1.77600 2676.09

Is it possible to configure the tool to work against a set of memcache servers, rather than being limited to a single one?