redislabs / memtier_benchmark Goto Github PK
View Code? Open in Web Editor NEWNoSQL Redis and Memcache traffic generation and benchmarking tool.
License: GNU General Public License v2.0
NoSQL Redis and Memcache traffic generation and benchmarking tool.
License: GNU General Public License v2.0
When arbitrary command (simple set command) is used and run-count is set to be greater than 1, only the first run succeeds and subsequent runs (> 1) were found to be failing.
Redis server was replying with "-ERR unknown command" response.
Examining the TCP dump of the redis commands sent by memtier_benchmark suggests that the RESP was malformatted for runs > 1.
Test Setup
memtier_benchmark: master branch @f55b4b6
Redis: 5.0.7, standalone
Runs on the same host
To reproduce the issue
memtier_benchmark -s localhost -p 6379 --protocol=redis --clients=1 --run-count=3 --command="set __key__ __data__"
Log
[RUN #1] Preparing benchmark client...
[RUN #1] Launching threads now...
[RUN #1 74%, 1 secs] 4 threads: 29582 ops, 29573 (avg: 29573) ops/sec, 2.22MB/sec (avg: 2.22MB/sec), 0.13 (avg: 0.13) msec latency^M[RUN #1 100%, 1 secs] 0 threads: 40000 ops, 29573 (avg: 29831) ops/sec, 2.22MB/sec (avg: 2.24MB/sec), 0.13 (avg: 0.13) msec latency^M
[RUN #2] Preparing benchmark client...
[RUN #2] Launching threads now...
server 127.0.0.1:6379 handle error response: -ERR unknown command `*3 $3 set `, with args beginning with: `memtier-6263819`, `xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx`,
server 127.0.0.1:6379 handle error response: -ERR unknown command `*3 $3 set `, with args beginning with: `memtier-6263819`, `xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx`,
server 127.0.0.1:6379 handle error response: -ERR unknown command `*3 $3 set `, with args beginning with: `memtier-6263819`, `xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx`,
... Repeated ...
[RUN #2 100%, 1 secs] 0 threads: 40000 ops, 24478 (avg: 23277) ops/sec, 4.80MB/sec (avg: 4.57MB/sec), 0.14 (avg: 0.15) msec latency^M
[RUN #3] Preparing benchmark client...
[RUN #3] Launching threads now...
server 127.0.0.1:6379 handle error response: -ERR unknown command `*3 $13 *3 $3 set `, with args beginning with: `memtier-6263819`, `xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx`,
server 127.0.0.1:6379 handle error response: -ERR unknown command `*3 $13 *3 $3 set `, with args beginning with: `memtier-6263819`, `xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx`,
server 127.0.0.1:6379 handle error response: -ERR unknown command `*3 $13 *3 $3 set `, with args beginning with: `memtier-6263819`, `xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx`,
... Repeated ...
[RUN #3 100%, 1 secs] 0 threads: 40000 ops, 24587 (avg: 24110) ops/sec, 5.34MB/sec (avg: 5.24MB/sec), 0.14 (avg: 0.14) msec latency^M
RESP captured for run>1
=== Request ===
=== Response ===
I would add it to the list of pre-reqs
Hi,
I wanted to install memtier_benchmark tool but it returned to a torture.I followed the instructions in README file but lots of packages are not exists in Centos6.4.
I installed below packages
yum install autoconf (http://petio.org/tools/autoconf.html)
yum install automake (http://petio.org/tools/automake.html)
yum install libmemcached
yum install libmemcached
yum install libmemcached-devel
After retrying ./configure gave below result.
No package 'libevent' found
Consider adjusting the PKG_CONFIG_PATH environment variable if you
installed software in a non-standard prefix.
Alternatively, you may set the environment variables LIBEVENT_CFLAGS
and LIBEVENT_LIBS to avoid the need to call pkg-config.
See the pkg-config man page for more details.
Than I try to install libevent but I got the conflicts
Transaction Check Error:
file /usr/bin/event_rpcgen.py from install of libevent-devel-2.0.19-1.rhel6.x86_64 conflicts with file from package compat-libevent14-1.4.13-1.rhel6.x86_64
file /usr/lib64/libevent_core.a from install of libevent-devel-2.0.19-1.rhel6.x86_64 conflicts with file from package compat-libevent14-1.4.13-1.rhel6.x86_64
file /usr/lib64/libevent_core.so from install of libevent-devel-2.0.19-1.rhel6.x86_64 conflicts with file from package compat-libevent14-1.4.13-1.rhel6.x86_64
file /usr/lib64/libevent_extra.a from install of libevent-devel-2.0.19-1.rhel6.x86_64 conflicts with file from package compat-libevent14-1.4.13-1.rhel6.x86_64
file /usr/lib64/libevent_extra.so from install of libevent-devel-2.0.19-1.rhel6.x86_64 conflicts with file from package compat-libevent14-1.4.13-1.rhel6.x86_64
When I try to erase compat-libevent14 package it shows that it is a dependency of nfs-utils.
Removing:
compat-libevent14 x86_64 1.4.13-1.rhel6 @pgdg91 466 k
Removing for dependencies:
nfs-utils x86_64 1:1.2.3-36.el6 @base 977 k
nfs-utils-lib x86_64 1.1.5-6.el6 @base 149 k
So I just cancelled installation of memtier_benchmark.And if it is possible can you prepare a tutorial for a centos release?
Regards
0、network latency
The network latency is around 0.15ms。
1、redis-cli latency
wanghaiyang@db-ck-1c-1:~$ redis-cli -h redis-bmk.uyffb3.ng.0001.apne1.cache.amazonaws.com --latency
2、memtier_benchmark result
memtier_benchmark -s redis-bmk.uyffb3.ng.0001.apne1.cache.amazonaws.com -t 2 -c 5 --ratio=1:6 --test-time=120 --select-db=0 --hide-histogram -x 3 --expiry-range=60-90 --data-size=384
The latency of memtier_benchmark is less than redis-cli latency and network latency,why?
I locally benchmark a Redis server 5.0 with the following command:
memtier_benchmark --show-config -d 4096 --key-maximum=50000 --hide-histogram -t 8 --ratio=1:1 --key-pattern=P:P --run-count 10
So, I have 50K keys and values of size 4K. The maximum size of my Redis instance must be 50K*4KB ~= 200MB
.
However, with each next run of memtier-benchmark, my Redis instance size increases by 200MB. After 10 runs, Redis eats 10*200MB = 2GB
of RAM.
I assumed that Redis must replace the same keys and delete old values, so its size is always capped at ~200MB. Why do I see 2GB then? Is this the intended behavior of --run-count
?
the test is like this.
run memtier on redis pack DB with replication with default endpoint bind.
kill endpoint and master redis node, failover of endpoint works well and slave is promoted well. memtier, even if memtier does multiple run ( using -x parameters ) it will never connect to the new endpoint.
restart memtier works and connect correctly. it will be great if memtier can refresh its connection to the new endpoint automatically.
Hi,
When I've tried to run memtier with --random-data
flag, I've noticed that the generated data is not really random. So, I've looked at the code that allocates the value buffer -
Line 209 in fdd22d8
/dev/urandom
?Thanks,
Adir
The --test-time
option forces memtier_benchmark to finish after test-time
seconds.
However, I noticed that each client actually waits to receive a reply from a Redis server, then checks if test-time
elapsed, and then stops. In my case, I have many parallel clients and many stragglers, where a single Redis reply can take several seconds to come back to client.
Thus, even though I specify --test-time=10
, it takes significantly more than 10 seconds. This not only takes too long for one experiment run, but also skews the results. (Which may be the behavior the authors desired?..)
I believe that all clients must die immediately after test-time
seconds. I fixed this issue with the following trivial patch:
diff --git a/client.cpp b/client.cpp
index e827195..f074877 100755
--- a/client.cpp
+++ b/client.cpp
@@ -557,7 +557,18 @@ int client_group::prepare(void)
void client_group::run(void)
{
+ // exit event loop immediately after test_time seconds
+ if (m_config->test_time > 0) {
+ struct timeval endtime;
+ endtime.tv_sec = m_config->test_time;
+ endtime.tv_usec = 0;
+ event_base_loopexit(m_base, &endtime);
+ }
+
event_base_dispatch(m_base);
+
+ for (std::vector<client*>::iterator i = m_clients.begin(); i != m_clients.end(); i++)
+ (*i)->set_end_time();
}
unsigned long int client_group::get_total_bytes(void)
https://hub.docker.com/r/redislabs/memtier_benchmark
Current master Dockerfile builds 1.2.15 and Hub's is 1.2.8.
Perhaps setting an Automated Build would be better. It would build always master branch to :latest image tag and would create individual image tags for each additional git tag.
See https://docs.docker.com/docker-hub/builds/
I'm trying to compile it on FreeBSD and get an error:
make all-am
CXX memtier_benchmark-memtier_benchmark.o
CXX memtier_benchmark-client.o
CXX memtier_benchmark-cluster_client.o
CXX memtier_benchmark-shard_connection.o
CXX memtier_benchmark-run_stats_types.o
In file included from run_stats_types.cpp:21:
In file included from ./run_stats_types.h:22:
In file included from ./memtier_benchmark.h:23:
./config_types.h:87:5: error: unknown type name 'pthread_mutex_t'; did you mean 'pthread_mutex'?
pthread_mutex_t m_mutex;
^~~~~~~~~~~~~~~
pthread_mutex
/usr/include/stdio.h:154:9: note: 'pthread_mutex' declared here
struct pthread_mutex _fl_mutex; / used for MT-safety */
^
In file included from run_stats_types.cpp:21:
In file included from ./run_stats_types.h:22:
In file included from ./memtier_benchmark.h:23:
./config_types.h:87:21: error: field has incomplete type 'pthread_mutex'
pthread_mutex_t m_mutex;
^
/usr/include/stdio.h:154:9: note: forward declaration of 'pthread_mutex'
struct pthread_mutex _fl_mutex; / used for MT-safety */
^
2 errors generated.
*** Error code 1
Stop.
I am trying to run memtier_benchmark on mac os. My Redis instances are running on my local machine which runs mac os
Trying with Docker
Running tests for a single instance(non cluster-mode) works fine after using -net=host. But when using --cluster-mode flag I am getting connection refused error. I guessed it is because CLUSTER SLOTS command will return ip as 127.0.0.1 from my local machine and then memtier uses it inside docker image and fails. To avoid that I set up rerouting of IP 127.0.0.1 to my local machines IP. This seemed working, but Redis started giving error "Error: Connection reset by peer" when using redis-cli from my docker image.
I gave up at this point
Building from source file
./configure
command gives an error at with this log
checking for pcre_compile in -lpcre... yes
checking for deflateInit_ in -lz... yes
checking for clock_gettime in -lrt... no
configure: error: rt is required for libevent
Can you please point me in the right direction?
Hi memtier_benchmark developers,
Currently memtier_benchmark can report average latency in the results, which is usually enough for simple benchmarks. it also provides the option to print the detail latency distribution. The data is aggregated through different latency level. However for storage/DB performance, people will also care about the tail latency(99%, 99.9% or 99.99%, example from fio). Is it possible to provide these tail latency data also?
ALL STATS
========================================================================
Type Ops/sec Hits/sec Misses/sec Latency KB/sec
------------------------------------------------------------------------
Sets 21536.09 --- --- 0.23100 22561.95
Gets 0.00 0.00 0.00 0.00000 0.00
Waits 0.00 --- --- 0.00000 ---
Totals 21536.09 0.00 0.00 0.23100 22561.95
thanks, -yuan
The memtier_benchmark
tool is using the -s
--server
as a parameter to pass the server address.
It would be good to be consistent with other redis-*
tools and use -h
--hostname
instead
When running the memtier_benchmark
client with either a couple more threads or a couple more connections than the default, at some point I got an accept4(): No file descriptors available
error and after some time the benchmark does not progress any more. Often this goes along with read error: Connection reset by peer
messages printed in the beginning.
Here an example for my local system and a custom Docker container:
$ docker run --rm -ti quay.io/kinvolk/memtier:latest-amd64
/ # memcached -u root &
/ # memtier_benchmark -p 11211 -P memcache_binary -t 100 --test-time=3 -x 1
[RUN #1] Preparing benchmark client...
accept4(): No file descriptors available
[RUN #1] Launching threads now...
[RUN #1 20%, 0 secs] 81 threads: 877668 ops, 1456138 (avg: 1439975) ops/sec, 55.07MB/sec (avg: 54.31MB/sec), 3.42 (avg: 3.40) msec latencyy
^C
[Stopped it at this point with Ctrl-C]
/ # memtier_benchmark -p 11211 -P memcache_binary -t 4 -c 400 --test-time=3 -x 1
[RUN #1] Preparing benchmark client...
accept4(): No file descriptors available
[RUN #1] Launching threads now...
[RUN #1 63%, 1 secs] 4 threads: 712807 ops, 0 (avg: 379203) ops/sec, 0.00KB/sec (avg: 14.30MB/sec), -nan (avg: 4.19) msec latencyy
^C [Similar problem here, stopped after some time]
root@simon-d14-1:~/benchmark/memtier_benchmark-1.2.7# ./configure
................
checking for gai_strerror in -lsocket... no
./configure: line 5344: syntax error near unexpected token LIBEVENT,' ./configure: line 5344:
PKG_CHECK_MODULES(LIBEVENT,'
root@simon-d14-1:~/benchmark/memtier_benchmark-1.2.7# cat /etc/*release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=15.10
It would be great if the rate limit (throttling) was configurable
I generated the file in the same format by referring to yours. Below is an example of the first few entries.
dumpflags, time, exptime, nbytes, nsuffix, it_flags, clsid, nkey, key, data
0, 0, 0, 200, 200, 0, 1, 200, sqjbmtzgladigkgckqxhqvkvnvcdrctjfqvsfnkwcocgagwvfbsbpadxxirdnkfexzifhmqxriaqvwiwxegholwgheabfqdhvxyqquaoxqrcebhjlvvsurgofgaapmqgyvtngtfkhiowfezfsjsyspzdffaojbwwqombzlvwhqqxdschnjgcjtgfmdxmvpmujreflbvc, ddcihrvrhfykwaprhhoxrgqcgcekmxkfkptrkhsvptkgnxqrazainyabxdkznmsgrpsgyeuxzkdopkpioevdmifibezsdhvglfjfojctdrxwlosunongwfpwrsuklfxrkneovbbhkspbmpsqloczpjzoiumspoxhtrqttsyhbrqgbhmibpdzpkboxrpshnylickresgq
0, 0, 0, 200, 200, 0, 1, 200, voaidlhyacrsxwurfwbdharbcuiotsaabyjhohqrperazwfialssbekpdhjujsfosvqvdtkvdocghakxqfxpbjrlpacixygnforgvpkojbylonzynvnezmhenlkmmerisipveqfrefpmgmnumtwtckdvlklegtcmygibekvktjmikqvobepdrdahcvildjprbgtlowam, fujwzhpvjutlhxycaivguimukkkhtvvkwktbtuhthudapksgynusznwnbaarjjqtdroukejmcgfdjzcerfjbbaoclgattmokezeeooptggpqdwagcltwqhbdnhsyxstxfdsgoemsbnbjyjgtmnklklhvjspsxbcnaffsofxrlxzgjwppdjyaioonzzxbhrsydivfmlkf
Note : The strings are not separated by newline they are just separated by the comma. There is a newline only after one complete entry
I want to insert these as key value pairs using memcache_text.
I am using the following command:
memtier_benchmark -p 11211 -P memcache_text -t 4 -n 100 –ratio 1000:0 -c 25 -x 5 --data-import=key.csv
Note: file name is key.csv
But I am getting this error-
key.csv:2: error parsing item values.
How to insert these key-value pairs using memtier_benchmark?
I am using memtier_benchmark to verify a private memcashed.
I would like to verify performance with diffrent loads, such as (70% get/30% read).
ideally with random keys and random values.
However I would like to ensure that hit ratio is 100%, i.e all gets are for keys already set.
How/If can i control this in memtier_benchmark?
memtier_benchmark -n 10 --cluster-mode -p 20000 --run-count 3
Thread 6 "memtier_benchma" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fffeffff700 (LWP 27658)]
0x000055555558a9bf in mbulk_size_el::get_next_mbulk (this=0x7fffe8001a50) at protocol.h:75
75 if (next->bulks_count == 0) {
(gdb) info registers
rax 0x30 48
rbx 0x7fffe8001b10 140737085709072
rcx 0x3 3
rdx 0x0 0
rsi 0x7fffe80008d0 140737085704400
rdi 0x7fffe8001a50 140737085708880
rbp 0x7fffefffec20 0x7fffefffec20
rsp 0x7fffefffec20 0x7fffefffec20
r8 0x7fffe80008d0 140737085704400
r9 0x0 0
r10 0x7fffe80008d0 140737085704400
r11 0x0 0
r12 0xe 14
r13 0x555555adbbc0 93824998030272
r14 0x7ffff7995d40 140737347411264
r15 0xe 14
rip 0x55555558a9bf 0x55555558a9bf <mbulk_size_el::get_next_mbulk()+27>
eflags 0x10206 [ PF IF RF ]
cs 0x33 51
ss 0x2b 43
ds 0x0 0
es 0x0 0
fs 0x0 0
gs 0x0 0
(gdb) bt
#0 0x000055555558a9bf in mbulk_size_el::get_next_mbulk (this=0x7fffe8001a50) at protocol.h:75
#1 0x0000555555588934 in redis_protocol::parse_response (this=0x55555654be20) at protocol.cpp:561
#2 0x0000555555571be8 in shard_connection::process_response (this=0x55555654bd90) at shard_connection.cpp:379
#3 0x0000555555570b84 in cluster_client_read_handler (bev=0x555555adb9a0, ctx=0x55555654bd90) at shard_connection.cpp:64
#4 0x00007ffff7995e32 in ?? () from /usr/lib/x86_64-linux-gnu/libevent-2.1.so.6
#5 0x00007ffff799b8f8 in ?? () from /usr/lib/x86_64-linux-gnu/libevent-2.1.so.6
#6 0x00007ffff799c33f in event_base_loop () from /usr/lib/x86_64-linux-gnu/libevent-2.1.so.6
#7 0x0000555555569dd7 in client_group::run (this=0x5555557c7140) at client.cpp:587
#8 0x000055555555cc08 in cg_thread_start (t=0x555555e888b0) at memtier_benchmark.cpp:996
#9 0x00007ffff700d6db in start_thread (arg=0x7fffeffff700) at pthread_create.c:463
#10 0x00007ffff63a088f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
When running ./configure
we get:
...
checking for clock_gettime in -lrt... no
configure: error: rt is required for libevent.
I am trying to work with Memcached, but I get "error: response parsing failed".
protocol.cpp:477: unsupported response: 'ERROR'.
error: response parsing failed.
hi there, running memtier with the Gaussian distribution option on macOS is stalling it ( build with TLS support is disabled ).
memtier_benchmark --hide-histogram --key-pattern=G:G
Output:
$ memtier_benchmark --hide-histogram --key-pattern=G:G
[RUN #1] Preparing benchmark client...
[RUN #1] Launching threads now...
[RUN #1 0%, 0 secs] 4 threads: 0 ops, 0 (avg: 0) ops/sec, 0.00KB/sec (avg: 0.00KB/sec), nan (avg: nan) msec latency
I'd like to verify the output data of a Memcached server (using --data-verify) using binary protocol. The --data-verify flag requires to import a file using --data-import.
I've tried creating a file following the structure described in README.import, but I keep getting an error.
Would someone be able to provide a sample file for --data-import, and what the expected output of memtier_benchmark would be when running --data-verify with the file?
Hi,
I am testing my middleware with for Multi-GETs in memcache-text
protocol
Acc. to memtier_benchmark --help
,
--multi-key-get=NUM Enable multi-key get commands, up to NUM keys (default: 0)
Let's say multi-key-get
is 5 in my case.
While running the experiments, I always see Multi-GETs with exactly 5 keys.
While this is technically correct, can we always assume that we will get exactly NUM
keys in the
multi-get or this is not the expected behaviour?
Thanks
Hello,
Please check result of execution (in attachment).
Memtier statistics:
,"Gets":{
"Ops/sec": 50.20
,"Hits/sec": 43.70
,"Misses/sec": 6.50
,"Latency": 39.83
,"KB/sec": 46.10
}
There is only one thread and 50.2 requests per 1 second. 1 second it is 1000ms. Let's calculate how much time in ms spended on each request: 1000/50.2 = 19.92. I
1.zip
.e. in memtier 1 request takes 19.92ms (it time incudes: request generating, logging and all internal memtier's logic).
I think that latency should be less than 19.2, because it should be the time only for request, but we have Latency=39.83.
I think that, Latency * "Ops/sec" should be less than 1000. But we have 39.83*50.2=1999.466
Regards,
Minas
When I invoke memtier_benchmark
with the --unix-socket
option, it segfaults when the client is trying to connect to Redis. For some reason, it tries to invoke get_connection_info
on the server_addr
, but it is null
. Here is the full backtrace:
#0 0x00007ffff7dc4365 in pthread_mutex_lock () from /lib/ld-musl-x86_64.so.1
#1 0x000055555556b716 in server_addr::get_connect_info (this=0x0,
ci=ci@entry=0x7fffffffd190) at config_types.cpp:234
#2 0x000055555555ca6d in client::connect (this=0x7ffff7b27d60)
at client.cpp:199
#3 0x000055555555ca01 in client::prepare (this=<optimized out>)
at client.cpp:344
#4 0x000055555555d52e in client_group::prepare (this=0x7ffff7d71fa0)
at client.cpp:539
#5 0x000055555555b259 in cg_thread::prepare (this=0x7ffff7d71ea0,
this=0x7ffff7d71ea0) at memtier_benchmark.cpp:790
#6 run_benchmark (run_id=1, cfg=0x7fffffffe4f0, obj_gen=0x555555788c40)
at memtier_benchmark.cpp:842
#7 0x0000555555559c5c in main (argc=<optimized out>, argv=<optimized out>)
at memtier_benchmark.cpp:1160
Hi,
In run_benchmark() (memtier_benchmark.cpp), duration is calculated as following:
float factor = ((float)(thread_counter - 1) / thread_counter);
duration = factor * duration + (float)(*i)->m_cg->get_duration_usec() / thread_counter ;
I would like to understand which theory the above formula is based on in order to obtain the run duration of multiple threads?
Any insight will be very much appreciated!
Thanks.
how to run it in multiple docker container cluster
$ for p in seq 1 10
;do /apps/svr/redis/bin/redis-cli -h 10.201.111.111 -p 6379 -a c4uT62vgCajbnvtgYFbq flushall ;./memtier_benchmark -s 10.201.111.111 -p 6379 -a c4xxxxxxxxxbq -n 10000 -c 16 -t 32 --reconnect-interval=10 | head -n 15 ;done
a lot of 'connect: connection failed: Connection timed out' error information in benchmark outputs.
-d --data-size=SIZE Object data size (default: 32)
What are the units for data size? Bytes?
From looking at memtier_benchmark's source code it didn't appear to support TLS/SSL connections to Redis. Are there any plans to implement this?
The updated new version does not seem to have the Makefile for running make command that is present in the README.
$ make
make: *** No targets specified and no makefile found. Stop
I can't run memtier_benchmark using Docker
docker run --rm memtier_benchmark -s localhost -p 11222 -network networkname --requests 4
(error: requests must be greater than zero.)
Not sure what I'm doing wrong...
In my test enviroment, I got the following outputs:
$ ./memtier_benchmark -s 192.168.111.126 -p 2001 -n 1 -c 1 -t 2 -d 20000 --cluster-mode
[RUN #1] Preparing benchmark client...
[RUN #1] Launching threads now...
[RUN #1 100%, 0 secs] 2 threads: 2 ops, 6557 (avg: 6557) ops/sec, 125.38MB/sec (avg: 125.38MB/sec), 0.39 (avg: 0.39) msec latency
It will hang here like forever. By reading the code, it seems it should exit and output like this:
[RUN #1] Preparing benchmark client...
[RUN #1] Launching threads now...
[RUN #1 100%, 0 secs] 0 threads: 2 ops, 0 (avg: 6756) ops/sec, 0.00KB/sec (avg: 129.20MB/sec), 0.00 (avg: 0.37) msec latency
2 Threads
1 Connections per thread
1 Requests per client
Sets 6779.66 --- --- 0.00 0.00 0.37200 132746.29
Gets 0.00 0.00 0.00 0.00 0.00 0.00000 0.00
Waits 0.00 --- --- --- --- 0.00000 ---
Totals 6779.66 0.00 0.00 0.00 0.00 0.37200 132746.29
SET 0.350 50.00
SET 0.390 100.00
Why? My test environment is cluster mode.
I am running memcached in MAC and using docker to benchmark memcached running on my local MAC. it's using latest in the master branch
here is the command:
docker run --rm memtier_benchmark -P memcache_text -p 11211 -s host.docker.internal --key-pattern=S:S --ratio=0:1
cmd put is
➜ memtier_benchmark git:(master) docker run --rm memtier_benchmark -P memcache_text -p 11211 -s host.docker.internal --key-pattern=S:S --ratio=0:1
[RUN #1] Preparing benchmark client...
[RUN #1] Launching threads now...
[RUN #1 100%, 165 secs] 0 threads: 2000000 ops, 13225 (avg: 12067) ops/sec, 1.03MB/sec (avg: 963.70KB/sec), 15.14 (avg: 2164.05) msec latencyyycyy
4 Threads
50 Connections per thread
10000 Requests per client
ALL STATS
=========================================================================
Type Ops/sec Hits/sec Misses/sec Latency KB/sec
-------------------------------------------------------------------------
Sets 0.00 --- --- 0.00000 0.00
Gets 12066.21 12066.21 0.00 2164.04600 963.62
Waits 0.00 --- --- 0.00000 ---
Totals 12066.21 12066.21 0.00 2164.04600 963.62
some value for histogram output
GET 50.000 99.93
GET 51.000 99.94
GET 52.000 99.94
GET 53.000 99.94
GET 54.000 99.94
GET 55.000 99.95
GET 56.000 99.95
GET 57.000 99.95
GET 58.000 99.95
GET 59.000 99.95
GET 60.000 99.95
GET 4300000.000 100.00
However, I am always getting 4300000.000 at the very end for GET. (same for SET).
which is probably causing my avg latency pretty high.
Can someone explain why I am always getting 4300000
If multiple threads and random write options are provided, all the threads perform the operation to same set of keys.
This is because the random_generator initializes the state of the random number generator by providing the same seed. So, all the threads initialize the generator with the same seed thus end up creating same set of keys.
int ret = initstate_r(1, m_state_array, sizeof(m_state_array), &m_data_blob);
I think the seed should be separate across threads.
Hi,
Thanks for developing and maintaining the great tool!
I was trying this tool with memcached and encountered a lot of "Connection error: Connection timed out". These errors weren't quite surprising to me since my memcached server was instrumented by an intel pintool and thus expected to be very slow when handling connections. So I was wondering if there's any way I can increase the timeout value for the benchmark tool to connect to the server. I figured out there's a libmemcached option "MEMCACHED_BEHAVIOR_CONNECT_TIMEOUT" that seems relevant, but I have no idea how to use it in the benchmark tool.
Any help/thought would be appreciated!
While performing benchmarking on time bound scenarios with heavy data sizes in the range of 8MBs or higher and with higher pipelines of >=8, I am noticing assertion failure.
Example command:
./memtier_benchmark -s localhost -p 6379 --test-time 300 -d 8000000 --pipeline=8 --threads=1 -c 20 --key-pattern=S:S --ratio=1:1
Error:
memtier_benchmark: client.cpp:426: void client::handle_event(short int): Assertion `finished() == false' failed.), -nan (avg: 20648.63) msec
This does not happen when time constraint is removed with memtier ( even though I select higher data sizes and pipelines ).
Also visiting the source code reveals, while finishing the tests, its trying to see if the write-buffer has any bytes left and if it is, then this assertion is thrown. Is this an intentional logic to show that benchmarking did not run clean when the time was fixed.
System details:
Intel DL 380 (Ubuntu 16.04 with default kernel)
RAM: 256GB , free ~247GB
The throughput number(5272.41 KB/sec) in the final table is incorrect.
$./memtier_benchmark --protocol redis -x 1 -s 209.243.160.17 -p 6379 --ratio=1:0 -t16 -c4 --key-pattern=R:R --key-minimum=1 --key-maximum=100000000 -n 15000 -d 32768[RUN #1] Preparing benchmark client...
[RUN #1] Launching threads now...
[RUN #1 100%, 266 secs] 0 threads: 960000 ops, 3549 (avg: 3604) ops/sec, 111.09MB/sec (avg: 112.81MB/sec), 18.08 (avg: 17.73) msec latency
16 Threads
4 Connections per thread
15000 Requests per thread
Type Ops/sec Hits/sec Misses/sec Latency KB/sec
Sets 3595.90 --- --- 17.72600 5272.41
Gets 0.00 0.00 0.00 0.00000 0.00
Totals 3595.90 0.00 0.00 17.72600 5272.41
Hi,
I am getting 0 hits/sec if i am running this benchmark on newly installed redis and memcache. Can you explain why this is so and how can i improve the hit percentage. Can you tell for what configurations what will be the result. Please help.
Thank you,
Kirti.
Similar to how redis-benchmark
works - would be extremely useful for module authors...
It does not occur when not using cluster_mode or when connecting to a local server.
It happened in the following environment.
bootjp@server:~$ memtier_benchmark --version
memtier_benchmark 1.2.17
Copyright (C) 2011-2017 Redis Labs Ltd.
This is free software. You may redistribute copies of it under the terms of
the GNU General Public License <http://www.gnu.org/licenses/gpl.html>.
There is NO WARRANTY, to the extent permitted by law.
bootjp@server:~$ uname -a
Linux server 4.15.0-72-generic #81-Ubuntu SMP Tue Nov 26 12:20:02 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
bootjp@server:~$ cat /etc/os-release
NAME="Ubuntu"
VERSION="18.04.3 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04.3 LTS"
VERSION_ID="18.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic
bootjp@server:~$
bootjp@server:~$ lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 24
On-line CPU(s) list: 0-23
Thread(s) per core: 2
Core(s) per socket: 6
Socket(s): 2
NUMA node(s): 2
Vendor ID: GenuineIntel
CPU family: 6
Model: 45
Model name: Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz
Stepping: 7
CPU MHz: 1676.846
CPU max MHz: 2500.0000
CPU min MHz: 1200.0000
BogoMIPS: 4000.00
Virtualization: VT-x
L1d cache: 32K
L1i cache: 32K
L2 cache: 256K
L3 cache: 15360K
NUMA node0 CPU(s): 0-5,12-17
NUMA node1 CPU(s): 6-11,18-23
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx lahf_lm epb pti ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid xsaveopt dtherm ida arat pln pts md_clear flush_l1d
Is there any other information I need?
I am trying to execute third party rest api from memtier it give below error while making the build
protocol.cpp:40:9: error: ‘SimpleWeb’ does not name a type
typedef SimpleWeb::ClientSimpleWeb::HTTP HttpClient;
So I am trying to use client_http.hpp to call rest api . But I am not able to create its instance.
I have include header file in protocol.cpp and in makefile
We've run into an interesting deadlock problem when implementing a middleware between memtier_benchmark and memched for a university project. We are three people with three independent implementations and we all have the same observation (see bold part below), so I'd like to ask you a few questions in order to gain more insights into memtier_benchmark and the deadlock problem.
We are to implement our middleware using a special architecture: a thread should accept connections and put these into a Java BlockingQueue
that is then processed by a number of worker threads who forward the request to the memcached server and finally reply to memtier_benchmark. There is a simple illustration for this on page 6 on the Project description. We all decided to serve a request from memtier_benchmark and then put the (still open) connection back into the queue.
The following works with any number of worker threads in the middleware. Here I will consider the case with a single worker thread serving memtier_benchmark requests. I use 1 memtier thread having 200 clients and releasing 10 requests of data size 16.
Starting memtier_benchmark. It works fine until it reaches about 95% progress, then everything blocks. Analysis showed that all three implementations block on reading on a connection from memtier_benchmark. For some reason, memtier_benchmark suddenly doesn't send any data on a given connection any more. This is the phenomenon we want to understand.
We have a possible explanation, but it's highly implementation specific for memtier_benchmark. This is why I'm writing this GitHub issue: we hope to get some light into this mystery. Our tentative explanation is as follows:
We make bold assumptions about memtier_benchmark's internals in this example. Would such a scenario be possible with memtier_benchmark? Do you have an idea why the read on a connected memtier_benchmark blocks? Thank you very much in advance, we are thrilled by the problem and looking forward to hear your input.
Steps to reproduce:
Tested platforms : x86 , QDF2400
Example command to run:
memtier_benchmark -s localhost -p 6379 -d 2 --pipeline=10 --threads=10 -c 50 --key-pattern=S:S --ratio=1:1 -n 10000 --out-file ./Memtier_out
PS : key-pattern=S:S ( i.e. sequential operations for read or write )
Hits/sec will always be 0 ( attached snapshot of this run MemtierIssue.jpg
)
This happens because as per the code change that has gone into 1.2.8+ affecting the client::create_request(struct timeval timestamp)
where the iter variable is not correctly updated for S operation get methods.
Introducing function : obj_iter_type(benchmark_config cfg, unsigned char index) in version 1.2.8+ to reduce redundant code for setting OBJECT_GENERATOR_ variable at everyplace,
For S operations, it always returns OBJECT_GENERATOR_KEY_SET_ITER and there is a NO assignment of iter = OBJECT_GENERATOR_KEY_GET_ITER for get method.
Hence, there is always a miss and hits/sec=0.
This happens for any key-pattern like S:S except R,G operations.
Attaching the patch for fixing this issue.
client.zip
Thanks,
Ashutosh Parida
Qualcomm Datacenter Technologies
the latency and the throughput are not matching.
the machines are free without any application process.
$ for p in seq 1 10
;do /apps/svr/redis/bin/redis-cli -h 10.201.111.111 -p 6379 -a c4uT62vgCajbnvtgYFbq flushall ;./memtier_benchmark -s 10.201.111.111 -p 6379 -a c4uT62vgCajbnvtgYFbq -n 5000 -c 32 -t 4 --reconnect-interval=5 | head -n 10 ;done
OK
[RUN #1] Preparing benchmark client...
[RUN #1] Launching threads now...
[RUN #1, 10 secs] 0 threads: 640000 ops, 62483 ops/sec, 2.32MB/sec, 0.70msec latency
4 Threads
32 Connections per thread
5000 Requests per thread
Type Ops/sec Hits/sec Misses/sec Latency KB/sec
Sets 5689.69 --- --- 0.70000 438.33
Gets 56834.37 0.00 56834.37 0.69700 1936.84
Totals 62524.06 0.00 56834.37 0.69700 2375.17
$ for p in seq 1 10
;do /apps/svr/redis/bin/redis-cli -h 10.201.111.111 -p 6379 flushall ;./memtier_benchmark -s 10.201.111.111 -p 6379 -n 5000 -c 32 -t 4 --reconnect-interval=5 | head -n 10 ;done
OK
[RUN #1] Preparing benchmark client...
[RUN #1] Launching threads now...
[RUN #1, 9 secs] 0 threads: 640000 ops, 69699 ops/sec, 2.59MB/sec, 1.78msec latency
4 Threads
32 Connections per thread
5000 Requests per thread
Type Ops/sec Hits/sec Misses/sec Latency KB/sec
Sets 6410.54 --- --- 1.77800 493.86
Gets 64034.96 0.00 64034.96 1.77500 2182.23
Totals 70445.50 0.00 64034.96 1.77600 2676.09
Is it possible to configure the tool to work against a set of memcache servers, rather than being limited to a single one?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.