tarantool / bench-run

Tarantool performance testing scripts

Tarantool

Tarantool is an in-memory computing platform consisting of a database and an application server.

It is distributed under BSD 2-Clause terms.

Key features of the application server:

• Heavily optimized Lua interpreter with incredibly fast tracing JIT compiler, based on LuaJIT 2.1.
• Cooperative multitasking, non-blocking IO.
• Persistent queues.
• Sharding.
• Cluster and application management framework.
• Access to external databases such as MySQL and PostgreSQL.
• A rich set of built-in and standalone modules.

Key features of the database:

• MessagePack data format and MessagePack based client-server protocol.
• Two data engines: 100% in-memory with complete WAL-based persistence and an own implementation of LSM-tree, to use with large data sets.
• Multiple index types: HASH, TREE, RTREE, BITSET.
• Document oriented JSON path indexes.
• Asynchronous master-master replication.
• Synchronous quorum-based replication.
• RAFT-based automatic leader election for the single-leader configuration.
• Authentication and access control.
• ANSI SQL, including views, joins, referential and check constraints.
• Connectors for many programming languages.
• The database is a C extension of the application server and can be turned off.

Supported platforms are Linux (x86_64, aarch64), Mac OS X (x86_64, M1), FreeBSD (x86_64).

Tarantool is ideal for data-enriched components of scalable Web architecture: queue servers, caches, stateful Web applications.

To download and install Tarantool as a binary package for your OS or using Docker, please see the download instructions.

To build Tarantool from source, see detailed instructions in the Tarantool documentation.

To find modules, connectors and tools for Tarantool, check out our Awesome Tarantool list.

Please report bugs to our issue tracker. We also warmly welcome your feedback on the discussions page and questions on Stack Overflow.

We accept contributions via pull requests. Check out our contributing guide.

Thank you for your interest in Tarantool!

bench-run's Issues

Fix the mess with benchmarks

At this point some scripts with tarantool's lua code are located in according repositories, and some are located at tarantool/bench-run. This is very missleading.

Not to mention extremely poor code quality in some of the benchmarks. All of this has to be refactored.

Create runner script for benchmarks to be run on local machines

At this moment benchmarks are run inside docker and are very dependent on prebuilt environment inside docker.

Fix TPCH failing benchmark

Right now tarantool's source code is being patched to pass through some of TPCH queries. This resulted in failure on one day. This patching has to stop.

Revive microb benchmark

Prepare scripts to be run on local machines

Part of #8

Track performance of tarantool/queue

Originally proposed in tarantool/queue#29

We'll have some benchmarks after tarantool/queue#139

Statistical analysis of benchmark results

The main concern about benchmarks on VMs is their instability. I have to run benchmarks multiple times and verify their stability/instability using statistical methods.

Benchmarks analyzer

At given state all benchmarks output their results in somehow readable format, but it is neither formalized, nor processable (yet). The script is required that is able to compare output of different benchmarks run.

It is supposed that every bench run can be exported as a .json file consisting of sequence of JSON-objects with the following schema: {"k":string,"v":number,"m":JSON-object}.

It is supposed that the result of every run can be stored for later analysis.
It is supposed that the result of every run can be checked against any other pre-stored result.

Benchmarks installer

Part of #8

Fix SQL schemas for benchmarks

Some of the benchmarks fail to run on newer Tarantool versions because of SQL incompatibility with older versions (TPCC for example, maybe some others).

Syntax mistake in tpc-h run script

Issue:

make: unrecognized option '--membind=1'
make: unrecognized option '--cpunodebind=1'
make: unrecognized option '--physcpubind=6,7,8,9,10,11'
Usage: make [options] [target] ...

Fix is:

diff --git a/benchs/tpch/run.sh b/benchs/tpch/run.sh
index b6fc512..b7dea48 100755
--- a/benchs/tpch/run.sh
+++ b/benchs/tpch/run.sh
@@ -13,14 +13,14 @@ killall tarantool 2>/dev/null || true
 sync && echo "sync passed" || echo "sync failed with error" $?
 echo 3 > /proc/sys/vm/drop_caches
 
-make bench-sqlite NUMAOPTS=$numaopts
+make bench-sqlite NUMAOPTS="$numaopts"
 make create_TNT_db
 
 killall tarantool 2>/dev/null || true
 sync && echo "sync passed" || echo "sync failed with error" $?
 echo 3 > /proc/sys/vm/drop_caches
 
-make bench-tnt NUMAOPTS=$numaopts
+make bench-tnt NUMAOPTS="$numaopts"
 make report
 sed "/-2/d" bench-tnt.csv | sed "s/;/:/" | sed "s/-1/0/" | tee tpc.h_result.txt