kataras / server-benchmarks

:rocket: Cross-platform transparent benchmarks for HTTP/2 Web Servers at 2020-2024

Server Benchmarks

A benchmark suite which, transparently, stress-tests web servers and generates a report in markdown. It measures the requests per second, data transferred and time between requests and responses.

Why YABS (Yet Another Benchmark Suite)

It's true, there already enough of benchmark suites to play around. However, most of them don't even contain real-life test applications to benchmark, therefore the results are not always accurate e.g. a route handler executes SQL queries or reads and sends JSON. This benchmark suite is a fresh start, it can contain any type of tests as the tests are running as self-executables and the measuring is done by a popular and trusted 3rd-party software which acts as a real HTTP Client (one more reason of transparency). Contributions and improvements are always welcomed here.

Use case

Measure the performance of application(s) between different versions or implementations (or web frameworks).

This suite can be further customized, through its tests.yml file, in order to test personal or internal web applications before their public releases.

Installation

The only requirement for the benchmark tool is the Go Programming Language.

$ go get github.com/kataras/server-benchmarks@master
$ go install github.com/codesenberg/bombardier@master

Depending on your test cases you may want to install Node.js and .NET Core too.

How to run

1. Navigate to your tests directory, the one which includes a tests.yml file
2. Open a terminal and execute: server-benchmarks
3. Wait for the executable server-benchmarks (or server-benchmarks.exe for windows) to finish
4. That's all, now open the the results from the generated RESULTS.md file.

Advanced usage

• Read the tests from the ./tests.dev.yml file
• Wait 3 seconds between tests
• Output the results to the ./dev directory
• Write the results to a remote google spreadsheet table, which you can convert to a graph later on (as shown above).

$ server-benchmarks --wait-run=3s -i ./tests.dev.yml -o ./dev -g-spreadsheet $GoogleSpreadsheetID -g-secret client_secret.json

Run using Docker

The only requirement is Docker.

$ docker run -v ${PWD}:/data kataras/server-benchmarks

Benchmarks

The following generated README contains benchmark results from builtin tests between popular HTTP/2 web frameworks as of 2022.

Note: it's possible that the contents of this file will be updated regularly to accept even more tests cases and frameworks.

System

Processor	AMD Ryzen 9 4900HS with Radeon Graphics
RAM	15.42 GB
OS	Microsoft Windows 11 Pro
Bombardier	v1.2.4
Go	go1.19beta1
.Net Core	6.0.300

| Node.js | v18.2.0 |

Last updated: Jun 20, 2022 at 8:17pm (UTC)

Terminology

Name is the name of the framework(or router) used under a particular test.

Reqs/sec is the avg number of total requests could be processed per second (the higher the better).

Latency is the amount of time it takes from when a request is made by the client to the time it takes for the response to get back to that client (the smaller the better).

Throughput is the rate of production or the rate at which data are transferred (the higher the better, it depends from response length (body + headers).

Time To Complete is the total time (in seconds) the test completed (the smaller the better).

Results

Test:Static

📖 Fires 1000000 requests, receives a static message as response.

Name	Language	Reqs/sec	Latency	Throughput	Time To Complete
Iris	Go	284059	438.34us	49.58MB	3.52s
Chi	Go	275525	451.01us	48.18MB	3.62s
Echo	Go	267815	466.16us	46.64MB	3.74s
Kestrel	C#	263479	472.72us	44.68MB	3.80s
Gin	Go	263399	472.70us	45.98MB	3.80s
Martini	Go	233051	534.43us	40.68MB	4.29s
Koa	Javascript	131274	0.93ms	29.24MB	7.50s
Buffalo	Go	78963	1.58ms	13.78MB	12.66s
Express	Javascript	41078	3.02ms	11.54MB	24.22s

Test:Parameterized

📖 Fires 550000 requests with a dynamic parameter of string, receives a hello text based on the parameter as response.

Name	Language	Reqs/sec	Latency	Throughput	Time To Complete
Iris	Go	277099	449.55us	53.07MB	1.99s
Chi	Go	272434	456.62us	52.21MB	2.02s
Echo	Go	261467	476.01us	50.14MB	2.10s
Gin	Go	259308	480.32us	49.70MB	2.12s
Kestrel	C#	233843	534.73us	43.34MB	2.36s
Martini	Go	225790	551.37us	43.29MB	2.44s
Koa	Javascript	114667	1.08ms	27.21MB	4.78s
Buffalo	Go	76747	1.63ms	14.71MB	7.17s
Express	Javascript	37110	3.32ms	11.11MB	14.69s

Test:REST

📖 Fires 200000 requests with a dynamic parameter of int, sends JSON as request body and receives JSON as response.

Name	Language	Reqs/sec	Latency	Throughput	Time To Complete
Iris	Go	238954	521.69us	64.15MB	0.84s
Gin	Go	229665	541.96us	62.86MB	0.87s
Chi	Go	228072	545.78us	62.61MB	0.88s
Echo	Go	224491	553.84us	61.70MB	0.89s
Martini	Go	198166	627.46us	54.47MB	1.01s
Kestrel	C#	163486	766.90us	47.42MB	1.23s
Buffalo	Go	102478	1.22ms	28.14MB	1.95s
Koa	Javascript	48425	2.56ms	15.39MB	4.14s
Express	Javascript	23622	5.25ms	9.04MB	8.41s

License

This project is licensed under the MIT License.

server-benchmarks's Issues

benchmark with Java framework

Golang's opponent should be Java, not node.js

Can someone write a benchmark test of the Java framework for the most comparison?

eg. SpringBoot/Vert.x/Spring Webflux

static express has issues ....

No matter if executed locally or via the docker image: it seems that the static express tests are failing ...

Any idea on this .....

THX a lot

➭ docker run -v ${PWD}:/data kataras/server-benchmarks
Running through docker container...[/go/src/server-benchmarks/_code/static/express]
$ npm install
$ node .
$ bombardier -c 125 -n 1000000 -t 2s -m GET --format=json --print=result http://localhost:5000
2020/05/20 09:19:36 Static:expressjs/express errors: dial tcp 127.0.0.1:5000: connect: connection refused

add fiber to the benchmark

Would you like to add Fiber to this benchmark?

docker run failed: 127.0.0.1:5000: connect: connection refused

Hello!

When I ran benchmarks in docker, i got error :

Static:expressjs/express errors: dial tcp 127.0.0.1:5000: connect: connection refused

How could I solve this problem? Thanks!