GithubHelp home page GithubHelp logo

giltene / wrk2 Goto Github PK

View Code? Open in Web Editor NEW
4.2K 69.0 376.0 16.3 MB

A constant throughput, correct latency recording variant of wrk

License: Apache License 2.0

Makefile 0.71% Lua 1.93% C 96.95% C++ 0.42%

wrk2's Introduction

wrk2

Build Status Gitter

a HTTP benchmarking tool based mostly on wrk

wrk2 is wrk modifed to produce a constant throughput load, and accurate latency details to the high 9s (i.e. can produce accurate 99.9999%'ile when run long enough). In addition to wrk's arguments, wrk2 takes a throughput argument (in total requests per second) via either the --rate or -R parameters (default is 1000).

CRITICAL NOTE: Before going farther, I'd like to make it clear that this work is in no way intended to be an attack on or a disparagement of the great work that Will Glozer has done with wrk. I enjoyed working with his code, and I sincerely hope that some of the changes I had made might be considered for inclusion back into wrk. As those of you who may be familiar with my latency related talks and rants, the latency measurement issues that I focused on fixing with wrk2 are extremely common in load generators and in monitoring code. I do not ascribe any lack of skill or intelligence to people who's creations repeat them. I was once (as recently as 2-3 years ago) just as oblivious to the effects of Coordinated Omission as the rest of the world still is.

wrk2 replaces wrk's individual request sample buffers with HdrHistograms. wrk2 maintains wrk's Lua API, including it's presentation of the stats objects (latency and requests). The stats objects are "emulated" using HdrHistograms. E.g. a request for a raw sample value at index i (see latency[i] below) will return the value at the associated percentile (100.0 * i / __len).

As a result of using HdrHistograms for full (lossless) recording, constant throughput load generation, and accurate tracking of response latency (from the point in time where a request was supposed to be sent per the "plan" to the time that it actually arrived), wrk2's latency reporting is significantly more accurate (as in "correct") than that of wrk's current (Nov. 2014) execution model.

It is important to note that in wrk2's current constant-throughput implementation, measured latencies are [only] accurate to a +/- ~1 msec granularity, due to OS sleep time behavior.

wrk2 is currently in experimental/development mode, and may well be merged into wrk in the future if others see fit to adopt it's changes.

The remaining part of the README is wrk's, with minor changes to reflect additional parameter and output. There is an important and detailed note at the end about about wrk2's latency measurement technique, including a discussion of Coordinated Omission, how wrk2 avoids it, and detailed output that demonstrates it.

wrk2 (as is wrk) is a modern HTTP benchmarking tool capable of generating significant load when run on a single multi-core CPU. It combines a multithreaded design with scalable event notification systems such as epoll and kqueue.

An optional LuaJIT script can perform HTTP request generation, response processing, and custom reporting. Several example scripts are located in scripts/

Basic Usage

wrk -t2 -c100 -d30s -R2000 http://127.0.0.1:8080/index.html

This runs a benchmark for 30 seconds, using 2 threads, keeping 100 HTTP connections open, and a constant throughput of 2000 requests per second (total, across all connections combined).

[It's important to note that wrk2 extends the initial calibration period to 10 seconds (from wrk's 0.5 second), so runs shorter than 10-20 seconds may not present useful information]

Output:

Running 30s test @ http://127.0.0.1:80/index.html
  2 threads and 100 connections
  Thread calibration: mean lat.: 9747 usec, rate sampling interval: 21 msec
  Thread calibration: mean lat.: 9631 usec, rate sampling interval: 21 msec
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     6.46ms    1.93ms  12.34ms   67.66%
    Req/Sec     1.05k     1.12k    2.50k    64.84%
  60017 requests in 30.01s, 19.81MB read
Requests/sec:   2000.15
Transfer/sec:    676.14KB

However, wrk2 will usually be run with the --latency flag, which provides detailed latency percentile information (in a format that can be easily imported to spreadsheets or gnuplot scripts and plotted per examples provided at http://hdrhistogram.org):

wrk -t2 -c100 -d30s -R2000 --latency http://127.0.0.1:80/index.html

Output:

Running 30s test @ http://127.0.0.1:80/index.html
  2 threads and 100 connections
  Thread calibration: mean lat.: 10087 usec, rate sampling interval: 22 msec
  Thread calibration: mean lat.: 10139 usec, rate sampling interval: 21 msec
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     6.60ms    1.92ms  12.50ms   68.46%
    Req/Sec     1.04k     1.08k    2.50k    72.79%
  Latency Distribution (HdrHistogram - Recorded Latency)
 50.000%    6.67ms
 75.000%    7.78ms
 90.000%    9.14ms
 99.000%   11.18ms
 99.900%   12.30ms
 99.990%   12.45ms
 99.999%   12.50ms
100.000%   12.50ms

Detailed Percentile spectrum:
     Value   Percentile   TotalCount 1/(1-Percentile)
     
     0.921     0.000000            1         1.00 
     4.053     0.100000         3951         1.11
     4.935     0.200000         7921         1.25
     5.627     0.300000        11858         1.43
     6.179     0.400000        15803         1.67
     6.671     0.500000        19783         2.00
     6.867     0.550000        21737         2.22
     7.079     0.600000        23733         2.50
     7.287     0.650000        25698         2.86
     7.519     0.700000        27659         3.33
     7.783     0.750000        29644         4.00
     7.939     0.775000        30615         4.44
     8.103     0.800000        31604         5.00
     8.271     0.825000        32597         5.71
     8.503     0.850000        33596         6.67
     8.839     0.875000        34571         8.00
     9.015     0.887500        35070         8.89
     9.143     0.900000        35570        10.00
     9.335     0.912500        36046        11.43
     9.575     0.925000        36545        13.33
     9.791     0.937500        37032        16.00
     9.903     0.943750        37280        17.78
    10.015     0.950000        37543        20.00
    10.087     0.956250        37795        22.86
    10.167     0.962500        38034        26.67
    10.279     0.968750        38268        32.00
    10.343     0.971875        38390        35.56
    10.439     0.975000        38516        40.00
    10.535     0.978125        38636        45.71
    10.647     0.981250        38763        53.33
    10.775     0.984375        38884        64.00
    10.887     0.985938        38951        71.11
    11.007     0.987500        39007        80.00
    11.135     0.989062        39070        91.43
    11.207     0.990625        39135       106.67
    11.263     0.992188        39193       128.00
    11.303     0.992969        39226       142.22
    11.335     0.993750        39255       160.00
    11.367     0.994531        39285       182.86
    11.399     0.995313        39319       213.33
    11.431     0.996094        39346       256.00
    11.455     0.996484        39365       284.44
    11.471     0.996875        39379       320.00
    11.495     0.997266        39395       365.71
    11.535     0.997656        39408       426.67
    11.663     0.998047        39423       512.00
    11.703     0.998242        39431       568.89
    11.743     0.998437        39439       640.00
    11.807     0.998633        39447       731.43
    12.271     0.998828        39454       853.33
    12.311     0.999023        39463      1024.00
    12.327     0.999121        39467      1137.78
    12.343     0.999219        39470      1280.00
    12.359     0.999316        39473      1462.86
    12.375     0.999414        39478      1706.67
    12.391     0.999512        39482      2048.00
    12.399     0.999561        39484      2275.56
    12.407     0.999609        39486      2560.00
    12.415     0.999658        39489      2925.71
    12.415     0.999707        39489      3413.33
    12.423     0.999756        39491      4096.00
    12.431     0.999780        39493      4551.11
    12.431     0.999805        39493      5120.00
    12.439     0.999829        39495      5851.43
    12.439     0.999854        39495      6826.67
    12.447     0.999878        39496      8192.00
    12.447     0.999890        39496      9102.22
    12.455     0.999902        39497     10240.00
    12.455     0.999915        39497     11702.86
    12.463     0.999927        39498     13653.33
    12.463     0.999939        39498     16384.00
    12.463     0.999945        39498     18204.44
    12.479     0.999951        39499     20480.00
    12.479     0.999957        39499     23405.71
    12.479     0.999963        39499     27306.67
    12.479     0.999969        39499     32768.00
    12.479     0.999973        39499     36408.89
    12.503     0.999976        39500     40960.00
    12.503     1.000000        39500          inf
#[Mean    =        6.602, StdDeviation   =        1.919]
#[Max     =       12.496, Total count    =        39500]
#[Buckets =           27, SubBuckets     =         2048]
----------------------------------------------------------
60018 requests in 30.00s, 19.81MB read
Requests/sec:   2000.28
Transfer/sec:    676.18KB

Scripting

wrk's public Lua API is:

init     = function(args)
request  = function()
response = function(status, headers, body)
done     = function(summary, latency, requests)

wrk = {
  scheme  = "http",
  host    = "localhost",
  port    = nil,
  method  = "GET",
  path    = "/",
  headers = {},
  body    = nil
}

function wrk.format(method, path, headers, body)

  wrk.format returns a HTTP request string containing the passed
  parameters merged with values from the wrk table.

global init     -- function called when the thread is initialized
global request  -- function returning the HTTP message for each request
global response -- optional function called with HTTP response data
global done     -- optional function called with results of run

The init() function receives any extra command line arguments for the script. Script arguments must be separated from wrk arguments with "--" and scripts that override init() but not request() must call wrk.init()

The done() function receives a table containing result data, and two statistics objects representing the sampled per-request latency and per-thread request rate. Duration and latency are microsecond values and rate is measured in requests per second.

latency.min              -- minimum value seen
latency.max              -- maximum value seen
latency.mean             -- average value seen
latency.stdev            -- standard deviation
latency:percentile(99.0) -- 99th percentile value
latency[i]               -- raw sample value

summary = {
  duration = N,  -- run duration in microseconds
  requests = N,  -- total completed requests
  bytes    = N,  -- total bytes received
  errors   = {
    connect = N, -- total socket connection errors
    read    = N, -- total socket read errors
    write   = N, -- total socket write errors
    status  = N, -- total HTTP status codes > 399
    timeout = N  -- total request timeouts
  }
}

Benchmarking Tips

The machine running wrk must have a sufficient number of ephemeral ports available and closed sockets should be recycled quickly. To handle the initial connection burst the server's listen(2) backlog should be greater than the number of concurrent connections being tested.

A user script that only changes the HTTP method, path, adds headers or a body, will have no performance impact. If multiple HTTP requests are necessary they should be pre-generated and returned via a quick lookup in the request() call. Per-request actions, particularly building a new HTTP request, and use of response() will necessarily reduce the amount of load that can be generated.

Acknowledgements

wrk2 is obviously based on wrk, and credit goes to wrk's authors for pretty much everything.

wrk2 uses my (Gil Tene's) HdrHistogram. Specifically, the C port written by Mike Barker. Details can be found at http://hdrhistogram.org . Mike also started the work on this wrk modification, but as he was stuck on a plane ride to New Zealand, I picked it up and ran it to completion.

wrk contains code from a number of open source projects including the 'ae' event loop from redis, the nginx/joyent/node.js 'http-parser', Mike Pall's LuaJIT, and the Tiny Mersenne Twister PRNG. Please consult the NOTICE file for licensing details.


A note about wrk2's latency measurement technique:

One of wrk2's main modification to wrk's current (Nov. 2014) measurement model has to do with how request latency is computed and recorded.

wrk's model, which is similar to the model found in many current load generators, computes the latency for a given request as the time from the sending of the first byte of the request to the time the complete response was received.

While this model correctly measures the actual completion time of individual requests, it exhibits a strong Coordinated Omission effect, through which most of the high latency artifacts exhibited by the measured server will be ignored. Since each connection will only begin to send a request after receiving a response, high latency responses result in the load generator coordinating with the server to avoid measurement during high latency periods.

There are various mechanisms by which Coordinated Omission can be corrected or compensated for. For example, HdrHistogram includes a simple way to compensate for Coordinated Omission when a known expected interval between measurements exists. Alternatively, some completely asynchronous load generators can avoid Coordinated Omission by sending requests without waiting for previous responses to arrive. However, this (asynchronous) technique is normally only effective with non-blocking protocols or single-request-per-connection workloads. When the application being measured may involve mutiple serial request/response interactions within each connection, or a blocking protocol (as is the case with most TCP and HTTP workloads), this completely asynchronous behavior is usually not a viable option.

The model I chose for avoiding Coordinated Omission in wrk2 combines the use of constant throughput load generation with latency measurement that takes the intended constant throughput into account. Rather than measure response latency from the time that the actual transmission of a request occurred, wrk2 measures response latency from the time the transmission should have occurred according to the constant throughput configured for the run. When responses take longer than normal (arriving later than the next request should have been sent), the true latency of the subsequent requests will be appropriately reflected in the recorded latency stats.

Note: This technique can be applied to variable throughput loaders. It requires a "model" or "plan" that can provide the intended start time if each request. Constant throughput load generators Make this trivial to model. More complicated schemes (such as varying throughput or stochastic arrival models) would likely require a detailed model and some memory to provide this information.

In order to demonstrate the significant difference between the two latency recording techniques, wrk2 also tracks an internal "uncorrected latency histogram" that can be reported on using the --u_latency flag. The following chart depicts the differences between the correct and the "uncorrected" percentile distributions measured during wrk2 runs. (The "uncorrected" distributions are labeled with "CO", for Coordinated Omission)

CO example

These differences can be seen in detail in the output provided when the --u_latency flag is used. For example, the output below demonstrates the difference in recorded latency distribution for two runs:

The first ["Example 1" below] is a relatively "quiet" run with no large outliers (the worst case seen was 11msec), and even wit the 99'%ile exhibit a ~2x ratio between wrk2's latency measurement and that of an uncorrected latency scheme.

The second run ["Example 2" below] includes a single small (1.4sec) disruption (introduced using ^Z on the apache process for simple effect). As can be seen in the output, there is a dramatic difference between the reported percentiles in the two measurement techniques, with wrk2's latency report [correctly] reporting a 99%'ile that is 200x (!!!) larger than that of the traditional measurement technique that was susceptible to Coordinated Omission.



Example 1: [short, non-noisy run (~11msec worst observed latency)]:

wrk -t2 -c100 -d30s -R2000 --u_latency http://127.0.0.1:80/index.html

Running 30s test @ http://127.0.0.1:80/index.html
  2 threads and 100 connections
  Thread calibration: mean lat.: 9319 usec, rate sampling interval: 21 msec
  Thread calibration: mean lat.: 9332 usec, rate sampling interval: 21 msec
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     6.18ms    1.84ms  11.31ms   69.23%
    Req/Sec     1.05k     1.11k    2.50k    64.57%
  Latency Distribution (HdrHistogram - Recorded Latency)
 50.000%    6.21ms
 75.000%    7.37ms
 90.000%    8.46ms
 99.000%   10.52ms
 99.900%   11.19ms
 99.990%   11.29ms
 99.999%   11.32ms
100.000%   11.32ms

  Detailed Percentile spectrum:
       Value   Percentile   TotalCount 1/(1-Percentile)

       0.677     0.000000            1         1.00
       3.783     0.100000         3952         1.11
       4.643     0.200000         7924         1.25
       5.263     0.300000        11866         1.43
       5.815     0.400000        15834         1.67
       6.207     0.500000        19783         2.00
       6.399     0.550000        21728         2.22
       6.639     0.600000        23702         2.50
       6.867     0.650000        25694         2.86
       7.095     0.700000        27664         3.33
       7.367     0.750000        29629         4.00
       7.499     0.775000        30615         4.44
       7.623     0.800000        31605         5.00
       7.763     0.825000        32599         5.71
       7.943     0.850000        33578         6.67
       8.183     0.875000        34570         8.00
       8.303     0.887500        35084         8.89
       8.463     0.900000        35566        10.00
       8.647     0.912500        36050        11.43
       8.911     0.925000        36559        13.33
       9.119     0.937500        37038        16.00
       9.279     0.943750        37289        17.78
       9.415     0.950000        37530        20.00
       9.559     0.956250        37776        22.86
       9.719     0.962500        38025        26.67
       9.919     0.968750        38267        32.00
      10.015     0.971875        38390        35.56
      10.103     0.975000        38514        40.00
      10.175     0.978125        38645        45.71
      10.239     0.981250        38772        53.33
      10.319     0.984375        38889        64.00
      10.375     0.985938        38953        71.11
      10.423     0.987500        39014        80.00
      10.479     0.989062        39070        91.43
      10.535     0.990625        39131       106.67
      10.591     0.992188        39199       128.00
      10.615     0.992969        39235       142.22
      10.631     0.993750        39255       160.00
      10.663     0.994531        39284       182.86
      10.703     0.995313        39318       213.33
      10.759     0.996094        39346       256.00
      10.823     0.996484        39363       284.44
      10.863     0.996875        39378       320.00
      10.919     0.997266        39392       365.71
      11.015     0.997656        39409       426.67
      11.079     0.998047        39423       512.00
      11.111     0.998242        39433       568.89
      11.127     0.998437        39439       640.00
      11.159     0.998633        39449       731.43
      11.167     0.998828        39454       853.33
      11.191     0.999023        39463      1024.00
      11.207     0.999121        39468      1137.78
      11.215     0.999219        39471      1280.00
      11.223     0.999316        39473      1462.86
      11.231     0.999414        39479      1706.67
      11.239     0.999512        39481      2048.00
      11.247     0.999561        39483      2275.56
      11.255     0.999609        39487      2560.00
      11.255     0.999658        39487      2925.71
      11.263     0.999707        39489      3413.33
      11.271     0.999756        39492      4096.00
      11.271     0.999780        39492      4551.11
      11.279     0.999805        39495      5120.00
      11.279     0.999829        39495      5851.43
      11.279     0.999854        39495      6826.67
      11.287     0.999878        39497      8192.00
      11.287     0.999890        39497      9102.22
      11.287     0.999902        39497     10240.00
      11.287     0.999915        39497     11702.86
      11.295     0.999927        39499     13653.33
      11.295     0.999939        39499     16384.00
      11.295     0.999945        39499     18204.44
      11.295     0.999951        39499     20480.00
      11.295     0.999957        39499     23405.71
      11.295     0.999963        39499     27306.67
      11.295     0.999969        39499     32768.00
      11.295     0.999973        39499     36408.89
      11.319     0.999976        39500     40960.00
      11.319     1.000000        39500          inf
#[Mean    =        6.178, StdDeviation   =        1.836]
#[Max     =       11.312, Total count    =        39500]
#[Buckets =           27, SubBuckets     =         2048]
----------------------------------------------------------

  Latency Distribution (HdrHistogram - Uncorrected Latency (measured without taking delayed starts into account))
 50.000%    2.68ms 
 75.000%    3.71ms
 90.000%    4.47ms
 99.000%    5.43ms
 99.900%    6.69ms
 99.990%    6.99ms
 99.999%    7.01ms
100.000%    7.01ms

  Detailed Percentile spectrum:
       Value   Percentile   TotalCount 1/(1-Percentile)

       0.264     0.000000            1         1.00
       1.111     0.100000         3954         1.11
       1.589     0.200000         7909         1.25
       1.970     0.300000        11852         1.43
       2.327     0.400000        15801         1.67
       2.679     0.500000        19751         2.00
       2.847     0.550000        21749         2.22
       3.003     0.600000        23703         2.50
       3.207     0.650000        25684         2.86
       3.483     0.700000        27664         3.33
       3.709     0.750000        29645         4.00
       3.813     0.775000        30623         4.44
       3.915     0.800000        31600         5.00
       4.035     0.825000        32591         5.71
       4.183     0.850000        33597         6.67
       4.319     0.875000        34580         8.00
       4.391     0.887500        35067         8.89
       4.471     0.900000        35561        10.00
       4.575     0.912500        36051        11.43
       4.683     0.925000        36545        13.33
       4.827     0.937500        37040        16.00
       4.903     0.943750        37296        17.78
       4.975     0.950000        37535        20.00
       5.035     0.956250        37779        22.86
       5.091     0.962500        38023        26.67
       5.159     0.968750        38281        32.00
       5.195     0.971875        38394        35.56
       5.231     0.975000        38520        40.00
       5.267     0.978125        38638        45.71
       5.311     0.981250        38767        53.33
       5.351     0.984375        38889        64.00
       5.375     0.985938        38957        71.11
       5.391     0.987500        39011        80.00
       5.415     0.989062        39076        91.43
       5.443     0.990625        39133       106.67
       5.519     0.992188        39193       128.00
       5.571     0.992969        39224       142.22
       5.671     0.993750        39254       160.00
       5.843     0.994531        39284       182.86
       5.915     0.995313        39315       213.33
       6.019     0.996094        39346       256.00
       6.087     0.996484        39362       284.44
       6.135     0.996875        39377       320.00
       6.323     0.997266        39392       365.71
       6.423     0.997656        39408       426.67
       6.471     0.998047        39423       512.00
       6.507     0.998242        39431       568.89
       6.535     0.998437        39439       640.00
       6.587     0.998633        39448       731.43
       6.643     0.998828        39454       853.33
       6.699     0.999023        39463      1024.00
       6.847     0.999121        39466      1137.78
       6.883     0.999219        39471      1280.00
       6.891     0.999316        39475      1462.86
       6.899     0.999414        39479      1706.67
       6.911     0.999512        39482      2048.00
       6.927     0.999561        39483      2275.56
       6.935     0.999609        39486      2560.00
       6.947     0.999658        39488      2925.71
       6.951     0.999707        39489      3413.33
       6.979     0.999756        39491      4096.00
       6.983     0.999780        39494      4551.11
       6.983     0.999805        39494      5120.00  
       6.983     0.999829        39494      5851.43
       6.987     0.999854        39496      6826.67
       6.987     0.999878        39496      8192.00
       6.987     0.999890        39496      9102.22
       6.995     0.999902        39497     10240.00
       6.995     0.999915        39497     11702.86
       7.007     0.999927        39499     13653.33
       7.007     0.999939        39499     16384.00
       7.007     0.999945        39499     18204.44
       7.007     0.999951        39499     20480.00
       7.007     0.999957        39499     23405.71
       7.007     0.999963        39499     27306.67
       7.007     0.999969        39499     32768.00
       7.007     0.999973        39499     36408.89
       7.015     0.999976        39500     40960.00
       7.015     1.000000        39500          inf
#[Mean    =        2.757, StdDeviation   =        1.254]
#[Max     =        7.012, Total count    =        39500]
#[Buckets =           27, SubBuckets     =         2048]
----------------------------------------------------------
  60031 requests in 30.01s, 19.82MB read
Requests/sec:   2000.67
Transfer/sec:    676.32KB


Example 2: [1.4 second ^Z artifact introduced on the httpd server]:

wrk -t2 -c100 -d30s -R2000 --u_latency http://127.0.0.1:80/index.html

Running 30s test @ http://127.0.0.1:80/index.html
  2 threads and 100 connections
  Thread calibration: mean lat.: 108237 usec, rate sampling interval: 1021 msec
  Thread calibration: mean lat.: 108178 usec, rate sampling interval: 1021 msec
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    63.66ms  223.37ms   1.42s    93.67%
    Req/Sec     1.00k   231.13     1.71k    89.47%
  Latency Distribution (HdrHistogram - Recorded Latency)
 50.000%    8.61ms
 75.000%   10.47ms
 90.000%   11.77ms
 99.000%    1.27s
 99.900%    1.42s
 99.990%    1.42s
 99.999%    1.42s
100.000%    1.42s

  Detailed Percentile spectrum:
       Value   Percentile   TotalCount 1/(1-Percentile)

       1.317     0.000000            1         1.00
       5.011     0.100000         3954         1.11
       6.215     0.200000         7903         1.25
       7.091     0.300000        11866         1.43
       7.827     0.400000        15810         1.67
       8.615     0.500000        19758         2.00
       8.991     0.550000        21734         2.22
       9.407     0.600000        23715         2.50
       9.871     0.650000        25713         2.86
      10.183     0.700000        27704         3.33
      10.471     0.750000        29648         4.00
      10.687     0.775000        30627         4.44
      10.903     0.800000        31604         5.00
      11.103     0.825000        32622         5.71
      11.295     0.850000        33583         6.67
      11.495     0.875000        34570         8.00
      11.615     0.887500        35067         8.89
      11.775     0.900000        35552        10.00
      12.047     0.912500        36048        11.43
      62.079     0.925000        36540        13.33
     294.399     0.937500        37054        16.00
     390.655     0.943750        37286        17.78
     524.799     0.950000        37525        20.00
     621.567     0.956250        37782        22.86
     760.831     0.962500        38062        26.67
     857.087     0.968750        38300        32.00
     903.679     0.971875        38399        35.56
     993.279     0.975000        38528        40.00
    1042.943     0.978125        38658        45.71
    1089.535     0.981250        38765        53.33
    1136.639     0.984375        38886        64.00
    1182.719     0.985938        38961        71.11
    1228.799     0.987500        39033        80.00
    1231.871     0.989062        39100        91.43
    1276.927     0.990625        39141       106.67
    1278.975     0.992188        39200       128.00
    1325.055     0.992969        39300       142.22
    1325.055     0.993750        39300       160.00
    1325.055     0.994531        39300       182.86
    1371.135     0.995313        39323       213.33
    1372.159     0.996094        39400       256.00
    1372.159     0.996484        39400       284.44
    1372.159     0.996875        39400       320.00
    1372.159     0.997266        39400       365.71
    1417.215     0.997656        39500       426.67
    1417.215     1.000000        39500          inf
#[Mean    =       63.660, StdDeviation   =      223.370]
#[Max     =     1416.192, Total count    =        39500]
#[Buckets =           27, SubBuckets     =         2048]
----------------------------------------------------------

  Latency Distribution (HdrHistogram - Uncorrected Latency (measured without taking delayed starts into account))
 50.000%    3.02ms
 75.000%    3.91ms
 90.000%    4.87ms
 99.000%    6.04ms
 99.900%    1.41s
 99.990%    1.41s
 99.999%    1.41s
100.000%    1.41s

  Detailed Percentile spectrum:
       Value   Percentile   TotalCount 1/(1-Percentile)

       0.325     0.000000            1         1.00
       1.210     0.100000         3950         1.11
       1.819     0.200000         7905         1.25
       2.343     0.300000        11851         1.43
       2.737     0.400000        15809         1.67
       3.015     0.500000        19760         2.00
       3.153     0.550000        21738         2.22
       3.289     0.600000        23722         2.50
       3.459     0.650000        25698         2.86
       3.691     0.700000        27650         3.33
       3.915     0.750000        29630         4.00
       4.053     0.775000        30621         4.44
       4.175     0.800000        31624         5.00
       4.299     0.825000        32612         5.71
       4.423     0.850000        33599         6.67
       4.587     0.875000        34564         8.00
       4.735     0.887500        35057         8.89
       4.871     0.900000        35560        10.00 
       4.975     0.912500        36051        11.43
       5.063     0.925000        36543        13.33 
       5.143     0.937500        37039        16.00
       5.187     0.943750        37282        17.78
       5.239     0.950000        37533        20.00
       5.291     0.956250        37782        22.86
       5.347     0.962500        38024        26.67
       5.435     0.968750        38278        32.00
       5.487     0.971875        38392        35.56
       5.555     0.975000        38514        40.00
       5.635     0.978125        38642        45.71
       5.735     0.981250        38760        53.33
       5.863     0.984375        38885        64.00
       5.899     0.985938        38946        71.11
       5.955     0.987500        39007        80.00
       6.019     0.989062        39071        91.43
       6.067     0.990625        39133       106.67
       6.127     0.992188        39194       128.00
       6.187     0.992969        39223       142.22
       6.287     0.993750        39255       160.00
       6.347     0.994531        39284       182.86
       6.411     0.995313        39315       213.33
       6.487     0.996094        39346       256.00
       6.539     0.996484        39362       284.44
       6.591     0.996875        39379       320.00
       6.711     0.997266        39392       365.71
    1413.119     0.997656        39411       426.67
    1414.143     0.998047        39500       512.00
    1414.143     1.000000        39500          inf
#[Mean    =        6.588, StdDeviation   =       70.891]
#[Max     =     1413.120, Total count    =        39500]
#[Buckets =           27, SubBuckets     =         2048]
----------------------------------------------------------
  60055 requests in 30.01s, 19.83MB read
Requests/sec:   2001.42
Transfer/sec:    676.57KB

wrk2's People

Contributors

anderender avatar blu2lz avatar ejc3 avatar giltene avatar gitter-badger avatar icflournoy avatar ickymettle avatar jfischoff avatar jmencak avatar jonrichards avatar markuskobler avatar mikeb01 avatar rrva avatar tobiasbueschel avatar trbenton avatar turbo87 avatar wg avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

wrk2's Issues

Catch up with upstream wrk

wrk2 was created in Nov. 2014 as an example of correcting coordinated omission in a load generator. It was basically a quick fork of wrk at the time, with minimal changes needed to achieve the purpose, created by @giletene and @mikeb01 as a result of a quick conversation at QCon SF.

The project turned out to be way more popular than I thought, or than originally intended. Wrk seems like a very solid base, but people looking for constant-rate capabilities and proper (not susceptible to coordinated omission) latency measurement seem to have picked up wrk2.

But since we had not put any real work into maintaining or enhancing wrk2 over the years, I'm sure wrk has added quite a bit in the 5 years since that we should simply "catch up on".

One simple way to do this is to follow Vizzini's directive and "go back to the beginning". Since applying the changes to wrk 5 years ago was "fairly simple" and since we had not strayed very far from the original work done in 2014, we can just pick up the latest/greatest wrk2, and apply the same logical changes to it that we did back then, to get an "up to date" wrk2 with all the wrk goodies. It took me and @mikeb01 only a few days to do it the first time, and applying it again "should" be even shorter... ;-)

I would prefer that we do this before starting to add any new features from PRs that have accumulated over the years, and any new features we want to add that are wrk2-specific (e.g. I'd really like to add a .hlog output support, which has long been part of hdrhistogram_c). Once we apply those PRs and additions, catching up with wrk will involve much more work...

Does anyone out there want to volunteer to do this "catch up to latest wrk" work?

Assertion failure while running benchmark

I get this error on running wrk2:

    Benchmark:
      ++++++++++++++++++++
      100Req/s Duration:300s open connections:20
      wrk2: src/hdr_histogram.c:54: counts_index: Assertion `bucket_index < h->bucket_count' failed.

I'm not sure why this is happening. If I run it again, it seems to work sometimes, but then fails again arbitrarily. I'm running this inside a docker container if that could be an issue.

Output format options

Hello! Are there any options to generate computer-readable reports? I.e. JSON, YAML or, at least, CSV :)

built version

Is there a built file and can put into my shell path?

or perhaps somewhere to learn how to build this project or lua projects in general.

Looks like a great tool, would like it in my terminal

Incorrect Reqs/Sec when using pipeline script

See the command used -->
./wrk -t1 -c1 -d1s -R1 --latency http://test.com/sample/hello.jsp -s pipeline_more.lua -- /100

Output

35 requests in 1.17s, 18.99KB read Requests/sec: 29.90 Transfer/sec: 16.22KB

Logs captured on the server shows the count as given below-->
`[test logs]$ wc -l localhost_access_log.2017-09-07.txt
60 localhost_access_log.2017-09-07.txt
[oqe@test logs]$

Also if we double the concurrent http connection in the above command ie, c=2 we get twice the request captured in the server logs as well as in the wrk data-->

./wrk -t1 -c2 -d1s -R1 --latency http://test.com/sample/hello.jsp -s pipeline_more.lua -- /100

Output

62 requests in 1.01s, 34.27KB read Requests/sec: 61.53 Transfer/sec: 34.01KB

Logs captured on the server shows the count as given below-->
[test logs]$ wc -l localhost_access_log.2017-09-07.txt
120 localhost_access_log.2017-09-07.txt
[oqe@test logs]$

This means that the wrk2 logging is capturing half of the request being made to the server.

Number of created connections sometimes is greater by 1 than specified at command line

Testing code (NodeJS 6)

const express = require('express');
const logger = require('log4js').getLogger();
const uid   = require('uid');
const assert = require('assert');
var app = express();

var reqCnt = {};
app.get('/', (req,res)=>{
    logger.debug(`Request, socket id=${req.connection.uid}`);
    assert.notEqual(reqCnt[req.connection.uid], undefined);
    reqCnt[req.connection.uid]++;
    res.send('ok');
});

setInterval( ()=>{
  var uids = Object.keys(reqCnt);
  //logger.debug(`Connection number: ${uids.length}`);
  for ( var uid of uids ) {
    //logger.debug(`${uid}: ${reqCnt[uid]} req/s`);
    reqCnt[uid] = 0;
  }
}, 1000);

var httpSrv = app.listen(8080);
httpSrv
    .on('connection', function(socket) {
      socket.uid = uid();
      reqCnt[socket.uid] = 0; // zero requests on that connection
      logger.debug(`connection is established: id=${socket.uid}, local: ${socket.localAddress}:${socket.localPort}, remote: ${socket.remoteAddress}:${socket.remotePort}`);
      logger.debug(socket.address());
      socket
      .on('data', function(data) {
          //logger.debug(`id=${socket.uid}, Data: unshown`);
      })
      .on('timeout', function() {
        logger.debug(`socket timeout: local: ${socket.localAddress}:${socket.localPort}, remote: ${socket.remoteAddress}:${socket.remotePort}`);
      })
      .on('close', function() {
        //console.log(socket);
        logger.debug(`socket closed: id=${socket.uid}, local: ${socket.localAddress}:${socket.localPort}, remote: ${socket.remoteAddress}:${socket.remotePort}`);
      });
    });

Running wrk2 with

./wrk -d 10 -t 2 -c 8 -R 20 http://localhost:8080

Output of

> node server.js | grep established
morpher@morpher-Inspiron-5447 ~/Documents/test_sockets $ node server.js   | grep established
[2016-08-22 09:57:14.123] [DEBUG] [default] - connection is established: id=wrpb37o, local: ::ffff:127.0.0.1:8080, remote: ::ffff:127.0.0.1:44399
[2016-08-22 09:57:14.161] [DEBUG] [default] - connection is established: id=wdy8o3m, local: ::ffff:127.0.0.1:8080, remote: ::ffff:127.0.0.1:44400
[2016-08-22 09:57:14.162] [DEBUG] [default] - connection is established: id=i5tqom2, local: ::ffff:127.0.0.1:8080, remote: ::ffff:127.0.0.1:44401
[2016-08-22 09:57:14.162] [DEBUG] [default] - connection is established: id=eedh7a2, local: ::ffff:127.0.0.1:8080, remote: ::ffff:127.0.0.1:44402
[2016-08-22 09:57:14.163] [DEBUG] [default] - connection is established: id=op3an49, local: ::ffff:127.0.0.1:8080, remote: ::ffff:127.0.0.1:44403
[2016-08-22 09:57:14.163] [DEBUG] [default] - connection is established: id=h1o8a37, local: ::ffff:127.0.0.1:8080, remote: ::ffff:127.0.0.1:44404
[2016-08-22 09:57:14.163] [DEBUG] [default] - connection is established: id=sqr6t8d, local: ::ffff:127.0.0.1:8080, remote: ::ffff:127.0.0.1:44405
[2016-08-22 09:57:14.164] [DEBUG] [default] - connection is established: id=unk79u6, local: ::ffff:127.0.0.1:8080, remote: ::ffff:127.0.0.1:44406
[2016-08-22 09:57:14.164] [DEBUG] [default] - connection is established: id=m7g0l5b, local: ::ffff:127.0.0.1:8080, remote: ::ffff:127.0.0.1:44407

There are 9 connections although I specified 8 connections.
Currently switched to apache benchmarking (ab) - it passes the test successfully.

send advanced http request and capture response using wrk/wrk2

Hi
I am trying to use wrk to soak my REST API interface in a product.
As part of the I am trying to send the wrk request for the below request (given at the bottom).
I am not able to set the –d options properly via json file or wrk array.
Will you be able to provide me proper wrk array format or Jason format.
I am NOT planning to use docker. As my server is not standalone web server. Its part of the product.

also I want to see the response to validate it.

Appreciate any help.
Thanks
M
Curl request taht want to send via wrk.
curl -k -i -d '{"jsonrpc":"2.0","id":101,"method":"ProvisionSubscriberInfo", "params":{"info": {"thing1": "@object_1"},"subId": 911}}' -X POST https://TPC-D7-07-001.phaedrus.sandvine.com/policyengine/v1/functions/ -H 'Authorization:

how to construct put requests using wrk or wrk2?

I want to load testing put request with different files (10000 files locally). How do I construct put requests? The closest I found is POST.lua, but how do I construct PUT request to upload files?
Thank you!

Problems when threads > rate

There seems to be an issue with wrk2 when number of threads specified (-t) is higher than rate (-R). wrk2 seems to be locking up.

Data discrepancy.Not getting correct data.Reqs/sec does not match the configuration

Check out the request which is being made-->
./wrk -t1 -c9 -d3s -R10 --latency http://test.com/sample/hello.jsp

Output-->
---------------------------------------------------------- 30 requests in 3.00s, 16.26KB read Requests/sec: 9.99 Transfer/sec: 5.41KB

As per the documentation, the total request made to the server should be 27?

Also in the below scenario-->
./wrk -t1 -c5 -d2s -R10 --latency http://test.com/sample/hello.jsp

16 requests in 2.03s, 8.67KB read
Requests/sec: 7.89
Transfer/sec: 4.28KB

How does the reqs/sec is >5?

Wrong latency distribution on the benchmark test

Dear all,
When I am using the wrk2 to do the http performance test, I found the latency distribution is quite puzzle as described below:

image
image

I have captured the http traffic with the wireshark, and I didn't find any traffic with latency greater than 1s.

image
image

WRK2 version: wrk 4.0.0 [kqueue] Copyright (C) 2012 Will Glozer
My commends: wrk2 -t 4 -c 10 -d 60s --rate 1000 -s header.lua --latency "$BaseUrl"

Can anyone tell me what's the problem here? Appreciated for your kindly help.

wrk2 parameters and their usage

Wrk2 is a little foreign to me vs other load test tools due to its mix of parameter options:

  • number of threads
  • number of connections
  • rate (throughput or requests/sec)

What are typical good combinations to use, and is there a relationship pattern to specifying those parameters? e.g. what if I want to double load or scale it by X?

With jmeter, gatling, and other tools, you just specify # of users or threads or connections. No need to work out allocation of threads vs connections combination. And with wrk2, then you need to add on rate to the mix, wonder how that plays in to the # of threads & connections for maintaining rate.

Wish these two tools provided more of a writeup on how you specify these parameters in relation to each other.

I assume the naive approach to be similar to jmeter and gatling would be to match all 2-3 parameters to each other? e.g. do 100 threads with 100 connections and 100 as the rate (for wrk2). Otherwise, a different combination is simply optimizing how the client/generator produces or distributes the load? The latter not my specialty in figuring out load generation optimization (whether we hit CPU/mem/fd/connection limits on the generator side, and which combination produces the max load, seems like much trial & error).

How do most people use wrk and wrk2 in regards to defining the parameters?

p100 corrected > p100 uncorrected

IIRC, a corrected histogram injects additional values between the higher observed value and the expected interval. So it makes sense that the p100 of the uncorrected histogram should be the same as the p100 in the corrected version. I've played with HDRHistogram in java and it does appear to behave as I expect, but wrk2 sometimes produces corrected p100s much higher than the uncorrected version.

Final few lines of the corrected version:
2306.047 1.000000 691130114 447392431.11
2306.047 1.000000 691130114 536870912.00
2306.047 1.000000 691130114 596523251.36
2306.047 1.000000 691130114 671088630.00
2308.095 1.000000 691130115 766958458.78
2308.095 1.000000 691130115 inf

[Mean = 1.335, StdDeviation = 4.240]

[Max = 2306.048, Total count = 691130115]

[Buckets = 27, SubBuckets = 2048]

Final few lines of the uncorrected version:
201.471 1.000000 691130114 383479229.39
201.471 1.000000 691130114 447392431.11
201.471 1.000000 691130114 536870912.00
201.471 1.000000 691130114 596523251.36
201.471 1.000000 691130114 671088630.00
203.775 1.000000 691130115 766958458.78
203.775 1.000000 691130115 inf

[Mean = 0.483, StdDeviation = 0.758]

[Max = 203.648, Total count = 691130115]

[Buckets = 27, SubBuckets = 2048]

[The command line used was: wrk -t5 -c32 -d24h -R8000 -U -sSCRIPT URL DATAFILE
version string: wrk 4.0.0 [epoll] Copyright (C) 2012 Will Glozer]

100% cpu

Has anyone else experienced wrk2's cpu usage shooting up to 100% after running for a while?

Unable to pull table from thread

Hi,
I am unable to pull table set in thread in done function. Content is added table in the response phase (response) function

I tried thread:get("table name").I get an empty table.

Where get info about format of output wrk2?

Sorry if this place not for this questipn. I dont found you email. I dont understand what is 1/(1-Percentile) formula in report (--latency). And what is "rate sampling interval" in thread calibration strings at beginning.

ssl context

seems like the code from the original (wrk) - does setup a ssl context in (wrk.c) but the implementation is incomplete as it does not load a cert

startup causing what appear to be false timeouts

To troubleshoot, I am using a server that returns after a 1 sec sleep. The payload of the response something like below where health.count is a Java AtomicInteger value that is incremented every time the resource is invoked:

{"status":"UP","customHealthCheck":{"status":"UP","health.count":8}

My wrk command line is:

wrk -s chip.lua -R 1 -t 1 -c 10 -d 30 http://localhost:8080/cc-auth-gateway/health

The lua script is simple:

function init(args)
requests = 0
responses = 0
wrk.method = 'GET'
wrk.headers["Api-Key"]="RTM";wrk.headers["User-Id"]="TEST";wrk.headers["Accept"]="application/json;v=1";wrk.headers["Content-Type"]="application/json;v=1";
end

function response(status, headers, body)
--print(status, body)
end

I put the line labeled "stopped" in the response_completed(). I put the "timeout" in check_timeout().
The "stopped" shows the diff between c->start and now which is way less than a sec. It also shows the body being the same for nearly all of the connections.

In addition the health.count is identical. If it had really called my server the health.count would have shown an incremental count value.

The server logs showed that the endpoint was called only once and not the 10 times that would have been expected.

..
..

Running 30s test @ http://localhost:8080/cc-auth-gateway/health
1 threads and 10 connections
stopped: conn: 0x7fe4c50020f8 now:1522155115404266 start:1522155115402504 diff:1762 status: 200 parser: 0x7fe4c5002100 body: 0x7fe4c5800400 body:{"status":"UP","customHealthCheck":{"status":"UP","health.count":13},"diskSpace"
stopped: conn: 0x7fe4c50041f0 now:1522155115409766 start:1522155115408162 diff:1604 status: 200 parser: 0x7fe4c50041f8 body: 0x7fe4c6000400 body:{"status":"UP","customHealthCheck":{"status":"UP","health.count":13},"diskSpace"
stopped: conn: 0x7fe4c50062e8 now:1522155115415657 start:1522155115413599 diff:2058 status: 200 parser: 0x7fe4c50062f0 body: 0x7fe4c6000c00 body:{"status":"UP","customHealthCheck":{"status":"UP","health.count":13},"diskSpace"
stopped: conn: 0x7fe4c50083e0 now:1522155115419798 start:1522155115418591 diff:1207 status: 200 parser: 0x7fe4c50083e8 body: 0x7fe4c5800c00 body:{"status":"UP","customHealthCheck":{"status":"UP","health.count":13},"diskSpace"
stopped: conn: 0x7fe4c500a4d8 now:1522155115424255 start:1522155115423011 diff:1244 status: 200 parser: 0x7fe4c500a4e0 body: 0x7fe4c5014e00 body:{"status":"UP","customHealthCheck":{"status":"UP","health.count":13},"diskSpace"
stopped: conn: 0x7fe4c500c5d0 now:1522155115429655 start:1522155115428230 diff:1425 status: 200 parser: 0x7fe4c500c5d8 body: 0x7fe4c5015200 body:{"status":"UP","customHealthCheck":{"status":"UP","health.count":13},"diskSpace"
stopped: conn: 0x7fe4c500e6c8 now:1522155115435156 start:1522155115433592 diff:1564 status: 200 parser: 0x7fe4c500e6d0 body: 0x7fe4c3013200 body:{"status":"UP","customHealthCheck":{"status":"UP","health.count":13},"diskSpace"
stopped: conn: 0x7fe4c50107c0 now:1522155115438764 start:1522155115437429 diff:1335 status: 200 parser: 0x7fe4c50107c8 body: 0x7fe4c3013a00 body:{"status":"UP","customHealthCheck":{"status":"UP","health.count":13},"diskSpace"
stopped: conn: 0x7fe4c50128b8 now:1522155115445146 start:1522155115443150 diff:1996 status: 200 parser: 0x7fe4c50128c0 body: 0x7fe4c6001400 body:{"status":"UP","customHealthCheck":{"status":"UP","health.count":13},"diskSpace"
stopped: conn: 0x7fe4c5000000 now:1522155116411484 start:1522155115397215 diff:1014269 status: 200 parser: 0x7fe4c5000008 body: 0x7fe4c6800800 body:{"status":"UP","customHealthCheck":{"status":"UP","health.count":14},"diskSpace"
timeout conn:0 connp: 0x7fe4c5000000 now:1522155117447507 start:1522155115397215 diff:2050292
timeout conn:1 connp: 0x7fe4c50020f8 now:1522155117447507 start:1522155115402504 diff:2045003
timeout conn:2 connp: 0x7fe4c50041f0 now:1522155117447507 start:1522155115408162 diff:2039345
timeout conn:3 connp: 0x7fe4c50062e8 now:1522155117447507 start:1522155115413599 diff:2033908

Improve aeTimeEvent resolution from 1 msec to 1 usec

One of the valid "nits" with wrk2 is that it can "over-report" latencies by up to 1msec because the rate-limiting model uses the call:
aeCreateTimeEvent(thread->loop, msec_to_wait, delay_request, c, NULL);
to wait before sending a request if "its time has not yet come". Because of the 1msec resolution of the ae async framework's aeTimeEvent and aeCreateTimeEvent, this can end up "oversleeping" by up to a millisecond, which ends up looking like a server problem when it is actually a load generator problem.

And the approach of "forgiving up to 1msec" is not a good one, as such an approach would miss real issues. IMO it is better to report pessimistic (could be somewhat worse than reality) latency numbers than ones that are better than reality.

But modern *nix variants can deal with clocks at a much finer resolution than 1msec (with e.g. nanosleep(), and timerfd), and the events should really be using a much finer resolution (e.g. 10-20 usec would not be unreasonable).

The really cool code in ae.c and friends appear to have originated from redis, and have not been touched in "forever". I'd like to work to improve the basic aeTimeEvent in that framework to include microsecond resolution information, along with a configurable quantum for actual time event resolution chunking.

The approach I'd take would probably keep the current external APIs (e.g. aeCreateTimeEvent which takes a 1-msec-unit time parameter) and all the current fields in e.g. aeTimeEvent (including when_sec and when_ms), but add an additional when_usec field for the optional microseconds-within-the-millisecond amount (defaults to 0) that some APIs may supply. We would then add additional APIs for those who want finer resolution (e.g. aeCreateTimeEventUsec(), aeWaitUsec(), aeGetTimeUsec()). We would change the underlying implementations that currently populate and use struct timevals (like aeProcessEvents(), aeApiPoll()), which already supports microsecond resolution, to correctly populate and usec-resolution information, and will use a timerfd to support sub-millisecond-resolution timing in epoll_wait() rather than rely on the timeout parameter.

The benefit of all this will be much more timely wakeups for delayed requests and a less pessimistic reporting of sub-millisecond response time levels, and better per-thread handling when >1000/sec/thread requests rates are actually possible.

Compile errors (dereferencing pointer to incomplete type) since wrk 4 backport

Hey,
I switched to the fork because I needed the -R option. Since commit 9e6583f (Backport of wrk 4 changes) it won't compile.
Error is "error: dereferencing pointer to incomplete type" for everything that's related to the struct addrinfo. It seems it's provided directly by lua. I tried to hunt it down but my time is limited at the moment. Now I work with the comment before the backport and it works just fine.
Thanks!

URL's with underscores are considered invalid.

Hi,

I noticed today that URLs with underscores are seen as invalid:

» wrk2 -c 100 -s test.lua -H "x-openrtb-version: 2.2" "http://smadexeast.rubicon.endpoints.ntoggle.com:8000/supply-partners/rubicon" -R 1000 -d 10
Running 10s test @ http://smadexeast.rubicon.endpoints.ntoggle.com:8000/supply-partners/rubicon
  2 threads and 100 connections
^C  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    34.39ms   16.09ms 242.56ms   85.53%
    Req/Sec       -nan      -nan   0.00      0.00%
  4147 requests in 4.25s, 4.49MB read
Requests/sec:    975.63
Transfer/sec:      1.06MB
[I] apenney at arya in ~
» wrk2 -c 100 -s test.lua -H "x-openrtb-version: 2.2" "http://smadex_east.rubicon.endpoints.ntoggle.com:8000/supply-partners/rubicon" -R 1000 -d 10
invalid URL: http://smadex_east.rubicon.endpoints.ntoggle.com:8000/supply-partners/rubicon
Usage: wrk <options> <url>
  Options:
    -c, --connections <N>  Connections to keep open
    -d, --duration    <T>  Duration of test
    -t, --threads     <N>  Number of threads to use

    -s, --script      <S>  Load Lua script file
    -H, --header      <H>  Add header to request
    -L  --latency          Print latency statistics
    -U  --u_latency        Print uncorrceted latency statistics
        --timeout     <T>  Socket/request timeout
    -B, --batch_latency    Measure latency of whole
                           batches of pipelined ops
                           (as opposed to each op)
    -v, --version          Print version details
    -R, --rate        <T>  work rate (throughput)
                           in requests/sec (total)
                           [Required Parameter]


  Numeric arguments may include a SI unit (1k, 1M, 1G)
  Time arguments may include a time unit (2s, 2m, 2h)
[I] apenney at arya in ~
»

The http_parser.c code seems to allow _'s so I'm not really sure where to start with fixing this. This was tested against master.

Startup issues a burst of requests causing timeouts

I am experiencing timeouts at the beginning of my test particularly as the number of threads is high and if the response time is in the 500 ms range. I created a small test and set threads to 10 and connection to 50 and the rate of 1 tps and forced my backend to response at 1 sec latency. I put a print() in my lua response() handler. What I notice is that all 50 connections send the request at startup which exceeds the 1 tps and since the response are so large it generates many timeouts.

confusing results when run with -H 'Connection: Close' for non-keepAlive http benchmark

Thanks for this excellent tools! We can use it get more accurate records about latency.
But when we try to use wrk2 do non-keepAlive http benchmark the results are confusing. e.g.

wrk2 -c 32  -t 16 -d 60s -R 60000 -H 'Connection: Close' http://127.0.0.1:8082
Running 1m test @ http://127.0.0.1:8082
  16 threads and 32 connections
  Thread calibration: mean lat.: 9223372036854776.000ms, rate sampling interval: 10ms
  Thread calibration: mean lat.: 9223372036854776.000ms, rate sampling interval: 10ms
  Thread calibration: mean lat.: 9223372036854776.000ms, rate sampling interval: 10ms
  Thread calibration: mean lat.: 9223372036854776.000ms, rate sampling interval: 10ms
  Thread calibration: mean lat.: 9223372036854776.000ms, rate sampling interval: 10ms
  Thread calibration: mean lat.: 9223372036854776.000ms, rate sampling interval: 10ms
  Thread calibration: mean lat.: 9223372036854776.000ms, rate sampling interval: 10ms
  Thread calibration: mean lat.: 9223372036854776.000ms, rate sampling interval: 10ms
  Thread calibration: mean lat.: 9223372036854776.000ms, rate sampling interval: 10ms
  Thread calibration: mean lat.: 9223372036854776.000ms, rate sampling interval: 10ms
  Thread calibration: mean lat.: 9223372036854776.000ms, rate sampling interval: 10ms
  Thread calibration: mean lat.: 9223372036854776.000ms, rate sampling interval: 10ms
  Thread calibration: mean lat.: 9223372036854776.000ms, rate sampling interval: 10ms
  Thread calibration: mean lat.: 9223372036854776.000ms, rate sampling interval: 10ms
  Thread calibration: mean lat.: 9223372036854776.000ms, rate sampling interval: 10ms
  Thread calibration: mean lat.: 9223372036854776.000ms, rate sampling interval: 10ms
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     -nanus    -nanus   0.00us    0.00%
    Req/Sec     0.00      0.00     0.00    100.00%
  0 requests in 1.00m, 2.05GB read
  Socket errors: connect 0, read 1675550, write 0, timeout 0
Requests/sec:      0.00
Transfer/sec:     35.01MB

For this real example it is a jetty web server which listens on port 8082.

Latency increases when threads > 1

Hi there,

I'm trying to understand why I would be seeing a jump in latency when the thread count is greater than 1.

On an c5.2xlarge ec2 instance with 4 physical CPUs (8 logical)

1 thread:

$ ./wrk -t1 -d30s -c100 -R100 http://myhost
Running 30s test @ http://myhost
  1 threads and 100 connections
  Thread calibration: mean lat.: 24.837ms, rate sampling interval: 57ms
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    23.23ms    4.27ms  49.63ms   84.20%
    Req/Sec    98.30     96.55   214.00     23.68%
  3001 requests in 30.02s, 832.31KB read
Requests/sec:     99.96
Transfer/sec:     27.72KB

4 threads:

$ ./wrk -t4 -d30s -c100 -R100 http://myhost
Running 30s test @ http://myhost
  4 threads and 100 connections
  Thread calibration: mean lat.: 189.007ms, rate sampling interval: 647ms
  Thread calibration: mean lat.: 341.843ms, rate sampling interval: 904ms
  Thread calibration: mean lat.: 336.302ms, rate sampling interval: 892ms
  Thread calibration: mean lat.: 343.629ms, rate sampling interval: 894ms
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   315.79ms  115.94ms 506.88ms   53.75%
    Req/Sec    24.92     10.10    38.00     70.83%
  2975 requests in 30.13s, 825.10KB read
Requests/sec:     98.74
Transfer/sec:     27.39KB

8 threads:

$ ./wrk -t8 -d30s -c100 -R100 http://myhost
Running 30s test @ http://myhost
  8 threads and 100 connections
  Thread calibration: mean lat.: 323.346ms, rate sampling interval: 902ms
  Thread calibration: mean lat.: 321.696ms, rate sampling interval: 901ms
  Thread calibration: mean lat.: 329.348ms, rate sampling interval: 895ms
  Thread calibration: mean lat.: 324.543ms, rate sampling interval: 912ms
  Thread calibration: mean lat.: 324.012ms, rate sampling interval: 907ms
  Thread calibration: mean lat.: 330.061ms, rate sampling interval: 910ms
  Thread calibration: mean lat.: 332.053ms, rate sampling interval: 914ms
  Thread calibration: mean lat.: 333.218ms, rate sampling interval: 910ms
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   335.85ms  129.11ms 488.96ms   79.56%
    Req/Sec    12.25      1.72    16.00     86.05%
  2994 requests in 30.06s, 830.37KB read
Requests/sec:     99.59
Transfer/sec:     27.62KB

I get similar results when I run the test for longer (5 mins). I observe the same latencies as measured by the server, so I wonder is there something different with the way multi-threaded load is generated?

Does `-c100 -R100` means "100 connections at 1 RPS", or "100 connections at random RPS" ?

I have a web app exposing an API which will be rate limited at 1 request per second per IP.
That being said I need to benchmark the maximum simultaneous users my app is able to absorb while keeping a P99 latency under 700ms.

In my understanding, wrk -t4 -c10000 -d70 -R 10000 -L http://localhost:8080 would translate to: using 4 threads, create 10.000 connections requesting localhost:8080 at 10.000 requests/s ALL TOGETHER, during 60 seconds (70-10s for initialization).

Is my understanding correct ? or will wrk2 RE-use some connections to issue more than 1 RPS in some of them ? for example, on the 10000 connections opened, is it possible that only 1 would actually be sending 10000 RPS while the 9999 others would stay opened without requests or something similar ?

EDIT
Just adding some illustration.
Does this wrk -t4 -c3 -d70 -R 3 -L http://localhost:8080 means:

option 1:

           |<  1 second  >|
client #1  |  1 request   |
client #2  |  1 request   |
client #3  |  1 request   |

or

option 2:

           |<  1 second  >|
client #1  |  2 request   |
client #2  |              |
client #3  |  1 request   |

wrk.lookup needs to gracefully handle errors

This came up testing a proxy that uses dns results to control load. When bandwidth reaches a software limit, dns queries switch to a "noroute" C record. The lua script would do a dns query every Nth request to see if the proxy had gone into denying requests.

The problem is that wrk.lookup immediately causes wrk 2 to terminate upon failing to get a connectable dns result.

It would be better if wrk.lookup would return a status code to indicate the lookup failure instead of terminating everything. I've currently hacked script.c to return a status code instead of terminating.

invalid option --R

Using the command from readme file:

wrk -t2 -c100 -d15s -R2000 http://127.0.0.1:8000/

Output:

wrk: invalid option -- R
Usage: wrk <options> <url>                            
  Options:                                            
    -c, --connections <N>  Connections to keep open   
    -d, --duration    <T>  Duration of test           
    -t, --threads     <N>  Number of threads to use   
                                                      
    -s, --script      <S>  Load Lua script file       
    -H, --header      <H>  Add header to request      
        --latency          Print latency statistics   
        --timeout     <T>  Socket/request timeout     
    -v, --version          Print version details      
                                                      
  Numeric arguments may include a SI unit (1k, 1M, 1G)
  Time arguments may include a time unit (2s, 2m, 2h)

Deadlock if -R is < -t

All in the title. The program freeze if -R is smaller than -t. I can imagine why it happen but that should be catched.

Failure to meet Rate

I do:
./wrk -t10 -c100 -d30s -R2000 --latency http://192.168.56.103/

however the rate is never more than 450 r/s

I can only achieve the desired rate for really low R values like 100 e.g.

The page I am hitting is static and fairly small (few Kb).

There are no socket errors or time-outs generated by the experiments.

I run ubuntu 16.4 with 4.4.0-31-generic kernel in vm with 4 cores and 10GB RAM.

Any chance that I have to enable or disable something for wrk2 to work properly?

Failed Installing wrk2 on Mac

I tried to install wrk2 on Mac (MacOS Mojave verison 10.14 (18A391)) via HomeBrew. Here is the output:

==> Installing wrk2 from jabley/wrk2
/usr/bin/sandbox-exec -f /private/tmp/homebrew20181011-62982-3usk4m.sb nice /usr/local/Homebrew/Library/Homebrew/vendor/portable-ruby/2.3.7/bin/ruby -W0 -I /usr/local/Homebrew/Library/Homebrew/vendor/bundle-standalone/bundler/../ruby/2.3.0/gems/ruby-macho-2.1.0/lib:/usr/local/Homebrew/Library/Homebrew/vendor/bundle-standalone/bundler/../ruby/2.3.0/gems/plist-3.4.0/lib:/usr/local/Homebrew/Library/Homebrew/vendor/bundle-standalone/bundler/:/usr/local/Homebrew/Library/Homebrew/vendor/bundle-standalone/bundler/../ruby/2.3.0/gems/backports-3.11.4/lib:/usr/local/Homebrew/Library/Homebrew/vendor/bundle-standalone/bundler/../ruby/2.3.0/gems/activesupport-5.2.1/lib:/usr/local/Homebrew/Library/Homebrew/vendor/bundle-standalone/bundler/../ruby/2.3.0/gems/tzinfo-1.2.5/lib:/usr/local/Homebrew/Library/Homebrew/vendor/bundle-standalone/bundler/../ruby/2.3.0/gems/thread_safe-0.3.6/lib:/usr/local/Homebrew/Library/Homebrew/vendor/bundle-standalone/bundler/../ruby/2.3.0/gems/minitest-5.11.3/lib:/usr/local/Homebrew/Library/Homebrew/vendor/bundle-standalone/bundler/../ruby/2.3.0/gems/i18n-1.1.0/lib:/usr/local/Homebrew/Library/Homebrew/vendor/bundle-standalone/bundler/../ruby/2.3.0/gems/concurrent-ruby-1.0.5/lib:/usr/local/Homebrew/Library/Homebrew/vendor/portable-ruby/2.3.7/lib/ruby/gems/2.3.0/gems/did_you_mean-1.0.0/lib:/usr/local/Homebrew/Library/Homebrew/vendor/portable-ruby/2.3.7/lib/ruby/site_ruby/2.3.0:/usr/local/Homebrew/Library/Homebrew/vendor/portable-ruby/2.3.7/lib/ruby/site_ruby/2.3.0/x86_64-darwin9.0:/usr/local/Homebrew/Library/Homebrew/vendor/portable-ruby/2.3.7/lib/ruby/site_ruby/2.3.0/universal-darwin9.0:/usr/local/Homebrew/Library/Homebrew/vendor/portable-ruby/2.3.7/lib/ruby/site_ruby:/usr/local/Homebrew/Library/Homebrew/vendor/portable-ruby/2.3.7/lib/ruby/vendor_ruby/2.3.0:/usr/local/Homebrew/Library/Homebrew/vendor/portable-ruby/2.3.7/lib/ruby/vendor_ruby/2.3.0/x86_64-darwin9.0:/usr/local/Homebrew/Library/Homebrew/vendor/portable-ruby/2.3.7/lib/ruby/vendor_ruby/2.3.0/universal-darwin9.0:/usr/local/Homebrew/Library/Homebrew/vendor/portable-ruby/2.3.7/lib/ruby/vendor_ruby:/usr/local/Homebrew/Library/Homebrew/vendor/portable-ruby/2.3.7/lib/ruby/2.3.0:/usr/local/Homebrew/Library/Homebrew/vendor/portable-ruby/2.3.7/lib/ruby/2.3.0/x86_64-darwin9.0:/usr/local/Homebrew/Library/Homebrew/vendor/portable-ruby/2.3.7/lib/ruby/2.3.0/universal-darwin9.0:/usr/local/Homebrew/Library/Homebrew -- /usr/local/Homebrew/Library/Homebrew/build.rb /usr/local/Homebrew/Library/Taps/jabley/homebrew-wrk2/wrk2.rb --verbose --HEAD
==> Cloning https://github.com/giltene/wrk2.git
Updating /Users/satrioadip/Library/Caches/Homebrew/wrk2--git
git config remote.origin.url https://github.com/giltene/wrk2.git
git config remote.origin.fetch \+refs/heads/master:refs/remotes/origin/master
git fetch origin
==> Checking out branch master
git checkout -f master --
Already on 'master'
Your branch is up to date with 'origin/master'.
git reset --hard origin/master
HEAD is now at e0109df Merge pull request #18 from alex-koturanov/master
cp -pR /Users/satrioadip/Library/Caches/Homebrew/wrk2--git/SCRIPTING /private/tmp/d20181011-62984-h35xj6/SCRIPTING
cp -pR /Users/satrioadip/Library/Caches/Homebrew/wrk2--git/LICENSE /private/tmp/d20181011-62984-h35xj6/LICENSE
cp -pR /Users/satrioadip/Library/Caches/Homebrew/wrk2--git/Makefile /private/tmp/d20181011-62984-h35xj6/Makefile
cp -pR /Users/satrioadip/Library/Caches/Homebrew/wrk2--git/NOTICE /private/tmp/d20181011-62984-h35xj6/NOTICE
cp -pR /Users/satrioadip/Library/Caches/Homebrew/wrk2--git/README.md /private/tmp/d20181011-62984-h35xj6/README.md
cp -pR /Users/satrioadip/Library/Caches/Homebrew/wrk2--git/.gitignore /private/tmp/d20181011-62984-h35xj6/.gitignore
cp -pR /Users/satrioadip/Library/Caches/Homebrew/wrk2--git/scripts/. /private/tmp/d20181011-62984-h35xj6/scripts
cp -pR /Users/satrioadip/Library/Caches/Homebrew/wrk2--git/deps/. /private/tmp/d20181011-62984-h35xj6/deps
cp -pR /Users/satrioadip/Library/Caches/Homebrew/wrk2--git/CoordinatedOmission/. /private/tmp/d20181011-62984-h35xj6/CoordinatedOmission
cp -pR /Users/satrioadip/Library/Caches/Homebrew/wrk2--git/.git/. /private/tmp/d20181011-62984-h35xj6/.git
cp -pR /Users/satrioadip/Library/Caches/Homebrew/wrk2--git/src/. /private/tmp/d20181011-62984-h35xj6/src
cp -pR /private/tmp/d20181011-62984-h35xj6/SCRIPTING /private/tmp/wrk2-20181011-62984-awtwi9/SCRIPTING
cp -pR /private/tmp/d20181011-62984-h35xj6/LICENSE /private/tmp/wrk2-20181011-62984-awtwi9/LICENSE
cp -pR /private/tmp/d20181011-62984-h35xj6/Makefile /private/tmp/wrk2-20181011-62984-awtwi9/Makefile
cp -pR /private/tmp/d20181011-62984-h35xj6/NOTICE /private/tmp/wrk2-20181011-62984-awtwi9/NOTICE
cp -pR /private/tmp/d20181011-62984-h35xj6/README.md /private/tmp/wrk2-20181011-62984-awtwi9/README.md
cp -pR /private/tmp/d20181011-62984-h35xj6/.gitignore /private/tmp/wrk2-20181011-62984-awtwi9/.gitignore
cp -pR /private/tmp/d20181011-62984-h35xj6/scripts/. /private/tmp/wrk2-20181011-62984-awtwi9/scripts
cp -pR /private/tmp/d20181011-62984-h35xj6/deps/. /private/tmp/wrk2-20181011-62984-awtwi9/deps
cp -pR /private/tmp/d20181011-62984-h35xj6/CoordinatedOmission/. /private/tmp/wrk2-20181011-62984-awtwi9/CoordinatedOmission
cp -pR /private/tmp/d20181011-62984-h35xj6/.git/. /private/tmp/wrk2-20181011-62984-awtwi9/.git
cp -pR /private/tmp/d20181011-62984-h35xj6/src/. /private/tmp/wrk2-20181011-62984-awtwi9/src
chmod -Rf +w /private/tmp/d20181011-62984-h35xj6
==> make
Building LuaJIT...
HOSTCC    host/minilua.o
HOSTCC    host/buildvm_asm.o
HOSTCC    host/buildvm_peobj.o
HOSTCC    host/buildvm_lib.o
HOSTCC    host/buildvm_fold.o
CC        lj_gc.o
CC        lj_char.o
CC        lj_obj.o
CC        lj_str.o
CC        lj_tab.o
CC        lj_func.o
CC        lj_udata.o
CC        lj_meta.o
CC        lj_debug.o
CC        lj_state.o
CC        lj_vmevent.o
CC        lj_vmmath.o
CC        lj_strscan.o
CC        lj_api.o
CC        lj_lex.o
CC        lj_parse.o
CC        lj_bcread.o
CC        lj_bcwrite.o
CC        lj_load.o
CC        lj_ir.o
CC        lj_opt_mem.o
CC        lj_opt_narrow.o
CC        lj_opt_dce.o
CC        lj_opt_loop.o
CC        lj_opt_split.o
CC        lj_opt_sink.o
CC        lj_mcode.o
CC        lj_snap.o
CC        lj_asm.o
CC        lj_trace.o
CC        lj_gdbjit.o
CC        lj_ctype.o
CC        lj_cdata.o
CC        lj_cconv.o
CC        lj_ccall.o
CC        lj_ccallback.o
CC        lj_carith.o
CC        lj_clib.o
CC        lj_cparse.o
CC        lj_lib.o
CC        lj_alloc.o
CC        lib_aux.o
CC        lib_package.o
CC        lib_init.o
CC        luajit.o
HOSTLINK  host/minilua
ld: library not found for -lgcc_s.10.4
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make[1]: *** [host/minilua] Error 1
make: *** [deps/luajit/src/libluajit.a] Error 2

==> Formula
Tap: jabley/wrk2
Path: /usr/local/Homebrew/Library/Taps/jabley/homebrew-wrk2/wrk2.rb
==> Configuration
HOMEBREW_VERSION: 1.7.7
ORIGIN: https://github.com/Homebrew/brew
HEAD: c54a657cd5987cba2718f7012a55101324fde8b1
Last commit: 3 days ago
Core tap ORIGIN: https://github.com/Homebrew/homebrew-core
Core tap HEAD: 683982a01204ddce1165daff00efc129d2e11adb
Core tap last commit: 12 hours ago
HOMEBREW_PREFIX: /usr/local
HOMEBREW_ENABLE_AUTO_UPDATE_MIGRATION: 1
CPU: octa-core 64-bit skylake
Homebrew Ruby: 2.3.7 => /usr/local/Homebrew/Library/Homebrew/vendor/portable-ruby/2.3.7/bin/ruby
Clang: 10.0 build 1000
Git: 2.17.1 => /Library/Developer/CommandLineTools/usr/bin/git
Curl: 7.54.0 => /usr/bin/curl
Java: 1.8.0_172
macOS: 10.14-x86_64
CLT: 10.0.0.0.1.1535735448
Xcode: N/A
==> ENV
HOMEBREW_CC: clang
HOMEBREW_CXX: clang++
MAKEFLAGS: -j8
CMAKE_PREFIX_PATH: /usr/local/opt/openssl:/usr/local
CMAKE_INCLUDE_PATH: /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/usr/include/libxml2:/Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/OpenGL.framework/Versions/Current/Headers
CMAKE_LIBRARY_PATH: /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/OpenGL.framework/Versions/Current/Libraries
PKG_CONFIG_PATH: /usr/local/opt/openssl/lib/pkgconfig
PKG_CONFIG_LIBDIR: /usr/lib/pkgconfig:/usr/local/Homebrew/Library/Homebrew/os/mac/pkgconfig/10.14
HOMEBREW_GIT: git
HOMEBREW_SDKROOT: /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk
ACLOCAL_PATH: /usr/local/share/aclocal
PATH: /usr/local/Homebrew/Library/Homebrew/shims/mac/super:/usr/local/opt/openssl/bin:/usr/bin:/bin:/usr/sbin:/sbin

Error: jabley/wrk2/wrk2 HEAD-e0109df did not build
Logs:
     /Users/satrioadip/Library/Logs/Homebrew/wrk2/01.make
     /Users/satrioadip/Library/Logs/Homebrew/wrk2/00.options.out
     /Users/satrioadip/Library/Logs/Homebrew/wrk2/01.make.cc
If reporting this issue please do so at (not Homebrew/brew or Homebrew/core):
https://github.com/jabley/homebrew-wrk2/issues

I also tried to build from source, here is the output:

Building LuaJIT...
HOSTCC    host/minilua.o
HOSTLINK  host/minilua
ld: warning: directory not found for option '-L/usr/local/Cellar/gsl/1.16/lib/'
ld: library not found for -lgcc_s.10.4
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make[1]: *** [host/minilua] Error 1
make: *** [deps/luajit/src/libluajit.a] Error 2

Is there any solution for this? Thank you.

latency vs u_latency

I dont understand the reasoning behind the recorded and uncorrected response times. I created a simple case where the server delays 1 sec.

I started wrk with 1 thread and 1 connection and with a target rate of 5 tps.

Obviously the rate cant be achieved but the response time is going to be 1 sec. When I run for 30 secs I find that the avg response was reported as 16 secs and the rate was indeed only 1 tps.

I dont understand why it would be 16 secs. The server is guaranteed to return in 1 sec and if I print out the u_latency it is indeed 1 sec. What is the reasoning behind the corrected values? We have a fork of this repo and we dump stats on intervals and the behavior is that the avg keeps forever increasing when the server cant keep up with the clients rate. I find that the value is not useful and is miss leading. The rate not meeting target is enough to tell me the server cant keep up.

I made a local change to the code to use u_latency vs latency for the summary stats and it is what I could expect to see. I would like an option select which to see in the summary status. Maybe an option for --u_latency and --latency and another for --details or something along that line. Is that something that would be of interest to you?

Same rate, different connections count - RS times greatly differs - my wrong setup?

Have simple http endpoint, I'm the only person using it (no noise from other users - did tests several times).
What I notice is that running 2 tests with same rate, but different client connections makes significant change in response times.

If number of connections would be too low, I would assume the rate requirement not fulfilled.

Maybe it has something to do with thread calibrating - I know proper performance testing is hard, but cannot find a guide of how to setup parameters (threads, rate, connections) and how it's connected with Thread calibrating - anyone knows what I'm doing wrong?

Details:

When I run this:
wrk -t5 -c5 -d1m -R10 --latency -s ./my_script.lua http://somewhere/something
I get this:

  5 threads and 5 connections
  Thread calibration: mean lat.: 197.251ms, rate sampling interval: 524ms
  Thread calibration: mean lat.: 196.915ms, rate sampling interval: 494ms
  Thread calibration: mean lat.: 204.040ms, rate sampling interval: 598ms
  Thread calibration: mean lat.: 181.220ms, rate sampling interval: 448ms
  Thread calibration: mean lat.: 191.452ms, rate sampling interval: 474ms
...
 50.000%  189.95ms
 75.000%  228.61ms
 90.000%  263.93ms
 99.000%  340.99ms
 99.900%  395.26ms
 99.990%  395.26ms
 99.999%  395.26ms
100.000%  395.26ms
...
600 requests in 1.00m, 6.30MB read
Requests/sec:     10.00

When I run this:
wrk -t5 -c10 -d1m -R10 --latency -s ./my_script.lua http://somewhere/something
I get this:

  5 threads and 10 connections
  Thread calibration: mean lat.: 303.611ms, rate sampling interval: 995ms
  Thread calibration: mean lat.: 353.468ms, rate sampling interval: 960ms
  Thread calibration: mean lat.: 316.724ms, rate sampling interval: 1056ms
  Thread calibration: mean lat.: 347.817ms, rate sampling interval: 1013ms
  Thread calibration: mean lat.: 330.926ms, rate sampling interval: 994ms
...
 50.000%  359.93ms
 75.000%  453.12ms
 90.000%  539.65ms
 99.000%  619.01ms
 99.900%  673.79ms
 99.990%  673.79ms
 99.999%  673.79ms
100.000%  673.79ms
600 requests in 1.00m, 6.30MB read
Requests/sec:     10.00

Should number of connections

Improvement - testing request

Hello,

I'm using wrk for testing of my apps regularly, and I find it awesome. One thing I'd like to be better is the testing request creation. I'm using the lua scripts for this, but could something like this be done?

Request:

echo -e 'GET /list HTTP/1.1\r\nContent-Type: application/json\r\nHost: localhost\r\nBODY DATA\r\n\r\n' | wrk http://localhost

This would allow me to craft any HTTP request (in any tool) and just copy&paste it to the terminal and be sure that wrk will create exactly this request for testing. Because sometimes when testing something behind Nginx proxy, instead of testing the app, you're actually testing the proxy, because it returns 302 Moved for some redirect. This is usually caused by incorrectly crafted request.

Do you think this would be possible, and worthwile doing?

Thanks!

Is it possible to access global data from request function

In the setup.lua example script, is it possible for all threads to be able to access the threads table from the init(), request(), and response() functions? When I try to do so currently, the table is empty and has a different address for each thread. It seems only the setup() and done() functions can access this memory.

support rate in minutes

I'd like to use wrk2 to test an API that downloads a large file, I'd need to set -R in requests per minute to a very low value, like -R5m. Is it possible now or can it be supported?

Can not build on OSX

First of all I tried make

~/src/wrk2 (master) → make
CC src/wrk.c
In file included from src/wrk.c:3:
src/wrk.h:11:10: fatal error: 'openssl/ssl.h' file not found
#include <openssl/ssl.h>
         ^
1 error generated.
make: *** [obj/wrk.o] Error 1

Ok, fine, we know how to fix that on mac

CFLAGS += -I/usr/local/include
 ~/src/wrk2 (master) → make
CC src/script.c
src/script.c:29:37: error: array has incomplete element type 'const struct luaL_reg'
static const struct luaL_reg addrlib[] = {
                                    ^
src/script.c:29:21: note: forward declaration of 'struct luaL_reg'
static const struct luaL_reg addrlib[] = {
                    ^
src/script.c:35:38: error: array has incomplete element type 'const struct luaL_reg'
static const struct luaL_reg statslib[] = {
                                     ^
src/script.c:29:21: note: forward declaration of 'struct luaL_reg'
static const struct luaL_reg addrlib[] = {
                    ^
src/script.c:41:39: error: array has incomplete element type 'const struct luaL_reg'
static const struct luaL_reg threadlib[] = {
                                      ^
src/script.c:29:21: note: forward declaration of 'struct luaL_reg'
static const struct luaL_reg addrlib[] = {
                    ^
src/script.c:53:5: warning: implicit declaration of function 'luaL_register' is invalid in C99 [-Wimplicit-function-declaration]
    luaL_register(L, NULL, addrlib);
    ^
src/script.c:109:20: warning: implicit declaration of function 'lua_objlen' is invalid in C99 [-Wimplicit-function-declaration]
    size_t count = lua_objlen(L, -1);
                   ^
2 warnings and 3 errors generated.
make: *** [obj/script.o] Error 1

I have no idea how to fix that. Could you please help?

Basic Usage Example Fails

This is the example given for basic usage:

wrk -t2 -c100 -d30s -R2000 http://127.0.0.1:8080/index.html

I believe wrk needs to be replaced with wrk2. At least, after installing this with homebrew, wrk2 was the name of the executable that got installed.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.