What server specs were used in this benchmark? about benchmarks

[Haaroon]

Ok so i used bombardier to test the async-graphql server, the starwars example, with a very simple query to get the name of id 1000, which is luke skywalker, and bombadier shows very impressive numbers of about 300k requests all HTTP 200. but these HTTP 200 are responses from the server, but the server has thus timed out and sends HTTP 200 with an error message inside of it.

I then wrote another benchmark with the "locust" python tool which does the same thing however it can check the result of the benchmark and my suspicious are indeed correct, there are plenty of failures. I ran this on an AWS box with 64 cores and 128GB of ram. Running the request remotely to the server. with 1000 users and a 100 user ramp up, after a few seconds it starts to fail.

I also then wrote yet another benchmark using artillery and again, the server starts to fall over.

In your benchmarking did you check the json response actually contained the correct data?

To add, have I done anything wrong? or am i looking at this the wrong way?

Seems im hitting tcp limits of the machine. but when adjusted i get much better results,, but stills caps around 5k a second

Edit: Clearly its an issue with the aws server i was usign, despite raising the ulimits to allow more open files it still does not allow a larger num of reqs, but if i use a bare metal server it has much more capacity.