alandzq / oltpbench-ii

OLTPBench

Benchmarking is incredibly useful, yet endlessly painful. This benchmark suite is the result of a group of Phd/post-docs/professors getting together and combining their workloads/frameworks/experiences/efforts. We hope this will save other people's time, and will provide an extensible platform, that can be grown in an open-source fashion.

OLTPBenchmark is a multi-threaded load generator. The framework is designed to be able to produce variable rate, variable mixture load against any JDBC-enabled relational database. The framework also provides data collection features, e.g., per-transaction-type latency and throughput logs.

Together with the framework we provide the following OLTP/Web benchmarks:

• TPC-C
• Wikipedia
• Synthetic Resource Stresser
• Twitter
• Epinions.com
• TATP
• AuctionMark
• SEATS ("Stonebraker Electronic Airline Ticketing System")
• YCSB
• JPAB (Hibernate)
• CH-benCHmark
• Voter (Japanese "American Idol")
• SIBench (Snapshot Isolation)
• SmallBank
• LinkBench

This framework is design to allow easy extension, we provide stub code that a contributor can use to include a new benchmark, leveraging all the system features (logging, controlled speed, controlled mixture, etc.)

Dependencies

• Java (+1.7)
• Apache Ant

Quick Start

See the on-line documentation on how to use OLTP-Bench.

Publications

If you are using this framework for your papers or for your work, please cite the paper:

OLTP-Bench: An extensible testbed for benchmarking relational databases D. E. Difallah, A. Pavlo, C. Curino, and P. Cudre-Mauroux. In VLDB 2014.

Also, let us know so we can add you to our list of publications.

OLTP-Bench II

Forked from https://github.com/oltpbenchmark/oltpbench with a focus on cleanup and modernization. Given the volume and scope of these changes, I have elected not to submit pull requests to the original project as it is unlikely they would or could be accepted. Please see Modifications to Original for changes in this fork.

See also: OLTP-Bench: An extensible testbed for benchmarking relational databases D. E. Difallah, A. Pavlo, C. Curino, and P. Cudre-Mauroux. In VLDB 2014.

Benchmarks

From Original Paper

• AuctionMark
• CH-benCHmark
• Epinions.com
• Resource Stresser
• SEATS
• SIBench
• SmallBank
• TATP
• TPC-C
• Twitter
• Voter
• Wikipedia
• YCSB

Added Later

• TPC-H
• TPC-DS - no configuration
• hyadapt - no configuration
• NoOp

Removed

• JPAB - this project appears abandoned and hasn't seen an update since 2012. I don't have a great deal of faith in a Hibernate benchmark that hasn't kept pace with Hibernate.
• LinkBench - no implementation

How to Build

Run the following command to build the distribution:

./mvnw clean package

The following files will be placed in the ./target folder, oltpbench2-x.y.z.tgz and oltpbench2-x.y.z.zip. Pick your poison.

The resulting .zip or .tgz file will have the following contents:

├── CONTRIBUTORS.md
├── LICENSE
├── README.md
├── config
│   ├── cockroachdb
│   │   ├── sample_auctionmark_config.xml
│   │   ├── sample_chbenchmark_config.xml
│   │   ├── sample_epinions_config.xml
│   │   ├── sample_noop_config.xml
│   │   ├── sample_resourcestresser_config.xml
│   │   ├── sample_seats_config.xml
│   │   ├── sample_sibench_config.xml
│   │   ├── sample_smallbank_config.xml
│   │   ├── sample_tatp_config.xml
│   │   ├── sample_tpcc_config.xml
│   │   ├── sample_tpcds_config.xml
│   │   ├── sample_tpch_config.xml
│   │   ├── sample_twitter_config.xml
│   │   ├── sample_voter_config.xml
│   │   ├── sample_wikipedia_config.xml
│   │   └── sample_ycsb_config.xml
│   ├── plugin.xml
│   └── postgres
│       └── ...
├── data
│   ├── tpch
│   │   ├── customer.tbl
│   │   ├── lineitem.tbl
│   │   ├── nation.tbl
│   │   ├── orders.tbl
│   │   ├── part.tbl
│   │   ├── partsupp.tbl
│   │   ├── region.tbl
│   │   └── supplier.tbl
│   └── twitter
│       ├── twitter_tweetids.txt
│       └── twitter_user_ids.txt
├── lib
│   └── ...
└── oltpbench2.jar

How to Run

Once you build and unpack the distribution, you can run oltpbench2 just like any other executable jar.

To bring up help contents:

java -jar oltpbench2.jar -h

To execute the tpcc benchmark:

java -jar oltpbench2.jar -b tpcc -c config/cockroachdb/sample_tpcc_config.xml --create=true --load=true --execute=true -s 5

For composite benchmarks like chbenchmark, which require multiple schemas to be created and loaded, you can provide a comma separated list: `

java -jar oltpbench2.jar -b tpcc,chbenchmark -c config/cockroachdb/sample_chbenchmark_config.xml --create=true --load=true --execute=true -s 5

The following options are provided:

usage: oltpbenchmark
 -b,--bench <arg>               [required] Benchmark class. Currently
                                supported: [tpcc, tpch, tatp, wikipedia,
                                resourcestresser, twitter, epinions, ycsb,
                                seats, auctionmark, chbenchmark, voter,
                                sibench, noop, smallbank, hyadapt]
 -c,--config <arg>              [required] Workload configuration file
    --clear <arg>               Clear all records in the database for this
                                benchmark
    --create <arg>              Initialize the database for this benchmark
 -d,--directory <arg>           Base directory for the result files,
                                default is current directory
    --dialects-export <arg>     Export benchmark SQL to a dialects file
    --execute <arg>             Execute the benchmark workload
 -h,--help                      Print this help
 -im,--interval-monitor <arg>   Throughput Monitoring Interval in
                                milliseconds
    --load <arg>                Load data using the benchmark's data
                                loader
 -s,--sample <arg>              Sampling window

How to see Postgres Driver logging

To enable logging for the PostgreSQL JDBC driver, add the following JVM property when starting...

-Djava.util.logging.config.file=src/main/resources/logging.properties

To modify the logging level you can update logging.properties

How to Release

./mvnw -B release:prepare
./mvnw -B release:perform

How to Add Support for a New Database

coming soon

Known Issues

Cockroach DB

My first priority is simply getting this code working against CockroachDB. No work has been put in to optimizing either the Database or the configurations for performance.

Benchmark	Config	Load	Run	Notes
auctionmark	✔️	✔️	✔️	issue #4, issue #40
chbenchmark	✔️	✔️	✔️	issue #5, issue #6
epinions	✔️	✔️	✔️	issue #7
hyadapt	❌			no config - issue #8
noop	✔️	✔️	✔️
resourcestresser	✔️	✔️	〰️	issue #41
seats	✔️	✔️	✔️	issue #10
sibench	✔️	✔️	✔️
smallbank	✔️	✔️	✔️
tatp	✔️	✔️	✔️
tpcc	✔️	✔️	✔️
tpcds	❌			no config - issue #11
tpch	✔️	✔️	✔️	issue #12
twitter	✔️	✔️	✔️	issue #13
voter	✔️	✔️	✔️
wikipedia	✔️	✔️	✔️
ycsb	✔️	✔️	✔️

Modifications to Original

This fork contains a number of significant structural modifications to the original project. This was done in an effort to clean up and modernize the code base, not to alter the spirit or function of the project. To this end, I did my best to leave the actual benchmark code functionally unchanged while improving where possible. My modifications are summarized below:

• Moved from Ant to Maven
  • Reorganized project to fit Maven structure
  • Removed static lib directory and dependencies
  • Updated required dependencies and removed unused or unwanted dependencies
  • Moved all non .java files to standard Maven resources directory
  • Shipped with Maven Wrapper
• Improved packaging and versioning
  • Moved to Calendar Versioning (https://calver.org/)
  • Project is now distributed as a .tgz or .zip with an executable .jar
  • All code updated to read resources from inside .jar instead of directory
• Built with and for Java 1.8 11
• Moved from direct dependence on Log4J to SLF4J
• Reorganized and renamed many files (mostly resources) for clarity and consistency
• Applied countless fixes based on "Static Analysis"
  • JDK migrations (boxing, un-boxing, etc.)
  • Implemented try-with-resources for all java.lang.AutoCloseable instances
  • Removed calls to printStackTrace() or System.out.println in favor of proper logging
• Reformatted code and cleaned up imports based on my preferences and using IntelliJ
• Removed all calls to assert... assert is disabled by default thus providing little real value while making the code incredibly hard to read and unnecessarily verbose
• Removed considerable amount of dead code, configurations, detritus and other nasty accumulations that didn't appear directly related to executing benchmarks
  • Removed IDE specific settings
  • Removed references to personal setups or cloud instances
  • Removed directories such as run, tools, nbproject, matlab, traces
  • Removed all references to JPAB benchmark, this project has not been updated since 2012
• Removed calls to commit() during Loader operations
• Refactored Worker and Loader usage of Connection objects and cleaned up transaction handling
• Introduced HikariCP as connection pool and DataSource instead of building connections from DriverManager as needed (default poolsize is 25 12)
• Introduced Dependabot to keep Maven dependencies up to date
• Removed upload, output, output-raw, output-samples timestamp, tracescript, histograms, run-script and verbose options. Those related to "output" were simply enabled by default.
• Refactored Catalog to be populated directly from the configured Benchmark database instead of proxied via HSQLDB. This eliminates the dependency on this project.