graphql-crystal / benchmarks

GraphQL server benchmarks

GraphQL server benchmarks

Graphql server benchmarks in many languages. Pull requests welcome, please read CONTRIBUTING.md

All servers implement a simple schema:

type Query {
  hello: String!
}

The returned string is always world.

The API is served over HTTP using a common web server and load tested using bombardier.

Results

Name	Language	Server	Latency avg	Requests
static-rust	Rust	Actix Web	1.88ms	110kps
graphql-crystal	Crystal	Kemal	3.34ms	60kps
gqlgen	Go	net/http	4.37ms	46kps
async-graphql	Rust	Actix Web	4.79ms	42kps
Juniper	Rust	Actix Web	5.43ms	37kps
Hot Chocolate	C#	ASP.NET	9.94ms	20kps
Mercurius	Node.js	Fastify	10.73ms	19kps
graphql-go	Go	net/http	11.45ms	17kps
GraphqlD	D	Vibe.d	12.40ms	16kps
agoo	Ruby/C	agoo	13.01ms	15kps
graphql-yoga	Node.js	http	16.08ms	13kps
graphql-jit	Node.js	http	21.31ms	9.4kps
Absinthe	Elixir	Phoenix	25.47ms	7.8kps
Hono	Bun	HonoJS	26.89ms	7.4kps
apollo	Node.js	Express	35.25ms	5.6kps
graphql-ruby	Ruby	Puma	48.30ms	5.3kps
nim-graphql	Nim	Chronos	57.11ms	3.6kps
graphql-js	Node.js	http	57.00ms	3.5kps
Sangria	Scala	Akka HTTP	112.40ms	1.8kps
Graphene	Python	gunicorn	128.63ms	1.5kps
Strawberry	Python	gunicorn	132.70ms	1.5kps

benchmarks's Issues

Dependency Dashboard

This issue provides visibility into Renovate updates and their statuses. Learn more

Open

These updates have all been created already. Click a checkbox below to force a retry/rebase of any.

• Update dependency org.sangria-graphql:sangria to v3

---

• Check this box to trigger a request for Renovate to run again on this repository

What server specs were used in this benchmark?

Thanks for benching alot of the gql tools, however its unclear what server was used to perform these benchmarks. I am looking to do my own benchmarking as a comparison. Thanks

Benchmark Elixir / Absinthe

It would be great if it could be added to the comparison.

Thank you so much for the benchmark! 🙌

Use codespaces for Benchmarks

Currently the benchmarks run on GitHub actions which doesn't guarantee consistent cpu across runs. Its better to integrate it witg codespaces or setup a dedicated ci machine with consistent cpu.

Benchmark for Sangria stuck

While raising the PR for new benchmark observing Sangria benchmark are getting stuck and eventually the Run action fail after 6Hr.

https://github.com/graphql-crystal/benchmarks/actions/runs/6730873449/job/18294413019

OCaml and Haskell examples

Will be interesting to see those two implementations.

For Haskell probably https://morpheusgraphql.com/

And for OCaml they have integration with Dream framework: https://ceramichacker.com/blog/29-3x-building-graphql-apis-with-dream

Warmup servers prior to executing benchmarks

Hi there 👋 I'm looking to add another Scala GraphQL library to the benchmarks. When I first looked at the current results I was a bit surprised that Sangria (Scala) was performing this badly. After looking at the benchmarking script, it seems that the servers are not warmed up prior to being benchmarked.

The JVM has a notoriously bad cold-start performance (see related post here), so any GraphQL libraries from JVM languages (Java, Kotlin, Scala, etc.) being benchmarked will show as having very poor performance if the server is not warmed up prior to measuring request latencies.

A very simple way to warmup the servers is to execute this line twice, discarding the results from the first invocation. Ideally, the first invocation should be with a smaller number of connections to prevent overwhelming the server as it warms up. I believe that this might also help resolve #217.

I'm happy to create a PR if you agree with this approach

GraphQL Yoga results

Hi!
This is Arda from @the-guild-org. I am one of maintainers of GraphQL Yoga, and we recently introduces a lot of performance improvements on GraphQL Yoga. I'd like to kindly ask if you can update the results based on the latest version of GraphQL Yoga's node package.

Thanks!

Benchmarks don't run

/nix/store/j2gmfdd29cp3rcvghmn1a5kgcnmf8myp-binutils-2.40/bin/ld: cannot find -lpcre2-8 (this usually means you need to install the development package for libpcre2-8): No such file or directory

Carry over to reality

Hi!

First of all thanks a lot for these benchmarks 🤗

When running my own with very small payloads - like say{ "data": { "user": { "name": "John Cena" } } } - I can see a 7.5x difference between graphql-js and async-graphql, which checks out pretty well with the benchmarks provided here.

However when I go ahead and create larger payloads (attached below) by creating new objects/structs like one would in a real world scenario (where they'd come from an ORM or whatever), the difference melts down to about 3.5x - which is still considerable, but largely different to the 7.5x (small) or even your 10x (your benchmark, even smaller) payloads. Do you think it might make sense to make the example payload larger to be more like what you'd see in production (at least partially, for lists), or am I overlooking something? Could it be that with bigger payloads more connections must be kept alive and streaming takes longer, so the advantages of faster execution are diminished? Something isn't quite clicking for me.

Traffic is about 86 MBps (for the bigger payload) which is under the 4 Gbps or 500 MBps network out the Linode Dedicated 4GB can do by quite a margin, so network shouldn't be the bottleneck. Bombardier is run on a separate Linode Shared 16GB so that shouldn't bottleneck either. Tests where run using both internet and VLAN.

Edit: In my benchmarks, the small response was 144 Bytes and the large one 11.3 KB, so ~80x difference.

Larger Payload

{
    "data": {
        "todos": [
            {
                "title": "Lorem ipsum dolor sit amet",
                "description": "Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam voluptua. At vero eos et accusam et justo duo dolores et ea rebum. Stet clita kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet. Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam voluptua. At vero eos et accusam et justo duo dolores et ea rebum. Stet clita kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet.",
                "watchers": [
                    {
                        "name": "John Cena"
                    },
                    {
                        "name": "John Cena"
                    },
                    {
                        "name": "John Cena"
                    },
                    {
                        "name": "John Cena"
                    },
                    {
                        "name": "John Cena"
                    }
                ]
            },
            {
                "title": "Lorem ipsum dolor sit amet",
                "description": "Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam voluptua. At vero eos et accusam et justo duo dolores et ea rebum. Stet clita kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet. Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam voluptua. At vero eos et accusam et justo duo dolores et ea rebum. Stet clita kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet.",
                "watchers": [
                    {
                        "name": "John Cena"
                    },
                    {
                        "name": "John Cena"
                    },
                    {
                        "name": "John Cena"
                    },
                    {
                        "name": "John Cena"
                    },
                    {
                        "name": "John Cena"
                    }
                ]
            },
            {
                "title": "Lorem ipsum dolor sit amet",
                "description": "Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam voluptua. At vero eos et accusam et justo duo dolores et ea rebum. Stet clita kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet. Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam voluptua. At vero eos et accusam et justo duo dolores et ea rebum. Stet clita kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet.",
                "watchers": [
                    {
                        "name": "John Cena"
                    },
                    {
                        "name": "John Cena"
                    },
                    {
                        "name": "John Cena"
                    },
                    {
                        "name": "John Cena"
                    },
                    {
                        "name": "John Cena"
                    }
                ]
            },
            {
                "title": "Lorem ipsum dolor sit amet",
                "description": "Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam voluptua. At vero eos et accusam et justo duo dolores et ea rebum. Stet clita kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet. Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam voluptua. At vero eos et accusam et justo duo dolores et ea rebum. Stet clita kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet.",
                "watchers": [
                    {
                        "name": "John Cena"
                    },
                    {
                        "name": "John Cena"
                    },
                    {
                        "name": "John Cena"
                    },
                    {
                        "name": "John Cena"
                    },
                    {
                        "name": "John Cena"
                    }
                ]
            },
            {
                "title": "Lorem ipsum dolor sit amet",
                "description": "Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam voluptua. At vero eos et accusam et justo duo dolores et ea rebum. Stet clita kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet. Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam voluptua. At vero eos et accusam et justo duo dolores et ea rebum. Stet clita kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet.",
                "watchers": [
                    {
                        "name": "John Cena"
                    },
                    {
                        "name": "John Cena"
                    },
                    {
                        "name": "John Cena"
                    },
                    {
                        "name": "John Cena"
                    },
                    {
                        "name": "John Cena"
                    }
                ]
            },
            {
                "title": "Lorem ipsum dolor sit amet",
                "description": "Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam voluptua. At vero eos et accusam et justo duo dolores et ea rebum. Stet clita kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet. Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam voluptua. At vero eos et accusam et justo duo dolores et ea rebum. Stet clita kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet.",
                "watchers": [
                    {
                        "name": "John Cena"
                    },
                    {
                        "name": "John Cena"
                    },
                    {
                        "name": "John Cena"
                    },
                    {
                        "name": "John Cena"
                    },
                    {
                        "name": "John Cena"
                    }
                ]
            },
            {
                "title": "Lorem ipsum dolor sit amet",
                "description": "Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam voluptua. At vero eos et accusam et justo duo dolores et ea rebum. Stet clita kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet. Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam voluptua. At vero eos et accusam et justo duo dolores et ea rebum. Stet clita kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet.",
                "watchers": [
                    {
                        "name": "John Cena"
                    },
                    {
                        "name": "John Cena"
                    },
                    {
                        "name": "John Cena"
                    },
                    {
                        "name": "John Cena"
                    },
                    {
                        "name": "John Cena"
                    }
                ]
            },
            {
                "title": "Lorem ipsum dolor sit amet",
                "description": "Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam voluptua. At vero eos et accusam et justo duo dolores et ea rebum. Stet clita kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet. Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam voluptua. At vero eos et accusam et justo duo dolores et ea rebum. Stet clita kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet.",
                "watchers": [
                    {
                        "name": "John Cena"
                    },
                    {
                        "name": "John Cena"
                    },
                    {
                        "name": "John Cena"
                    },
                    {
                        "name": "John Cena"
                    },
                    {
                        "name": "John Cena"
                    }
                ]
            },
            {
                "title": "Lorem ipsum dolor sit amet",
                "description": "Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam voluptua. At vero eos et accusam et justo duo dolores et ea rebum. Stet clita kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet. Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam voluptua. At vero eos et accusam et justo duo dolores et ea rebum. Stet clita kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet.",
                "watchers": [
                    {
                        "name": "John Cena"
                    },
                    {
                        "name": "John Cena"
                    },
                    {
                        "name": "John Cena"
                    },
                    {
                        "name": "John Cena"
                    },
                    {
                        "name": "John Cena"
                    }
                ]
            },
            {
                "title": "Lorem ipsum dolor sit amet",
                "description": "Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam voluptua. At vero eos et accusam et justo duo dolores et ea rebum. Stet clita kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet. Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam voluptua. At vero eos et accusam et justo duo dolores et ea rebum. Stet clita kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet.",
                "watchers": [
                    {
                        "name": "John Cena"
                    },
                    {
                        "name": "John Cena"
                    },
                    {
                        "name": "John Cena"
                    },
                    {
                        "name": "John Cena"
                    },
                    {
                        "name": "John Cena"
                    }
                ]
            },
            {
                "title": "Lorem ipsum dolor sit amet",
                "description": "Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam voluptua. At vero eos et accusam et justo duo dolores et ea rebum. Stet clita kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet. Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam voluptua. At vero eos et accusam et justo duo dolores et ea rebum. Stet clita kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet.",
                "watchers": [
                    {
                        "name": "John Cena"
                    },
                    {
                        "name": "John Cena"
                    },
                    {
                        "name": "John Cena"
                    },
                    {
                        "name": "John Cena"
                    },
                    {
                        "name": "John Cena"
                    }
                ]
            },
            {
                "title": "Lorem ipsum dolor sit amet",
                "description": "Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam voluptua. At vero eos et accusam et justo duo dolores et ea rebum. Stet clita kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet. Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam voluptua. At vero eos et accusam et justo duo dolores et ea rebum. Stet clita kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet.",
                "watchers": [
                    {
                        "name": "John Cena"
                    },
                    {
                        "name": "John Cena"
                    },
                    {
                        "name": "John Cena"
                    },
                    {
                        "name": "John Cena"
                    },
                    {
                        "name": "John Cena"
                    }
                ]
            },
            {
                "title": "Lorem ipsum dolor sit amet",
                "description": "Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam voluptua. At vero eos et accusam et justo duo dolores et ea rebum. Stet clita kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet. Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam voluptua. At vero eos et accusam et justo duo dolores et ea rebum. Stet clita kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet.",
                "watchers": [
                    {
                        "name": "John Cena"
                    },
                    {
                        "name": "John Cena"
                    },
                    {
                        "name": "John Cena"
                    },
                    {
                        "name": "John Cena"
                    },
                    {
                        "name": "John Cena"
                    }
                ]
            },
            {
                "title": "Lorem ipsum dolor sit amet",
                "description": "Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam voluptua. At vero eos et accusam et justo duo dolores et ea rebum. Stet clita kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet. Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam voluptua. At vero eos et accusam et justo duo dolores et ea rebum. Stet clita kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet.",
                "watchers": [
                    {
                        "name": "John Cena"
                    },
                    {
                        "name": "John Cena"
                    },
                    {
                        "name": "John Cena"
                    },
                    {
                        "name": "John Cena"
                    },
                    {
                        "name": "John Cena"
                    }
                ]
            },
            {
                "title": "Lorem ipsum dolor sit amet",
                "description": "Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam voluptua. At vero eos et accusam et justo duo dolores et ea rebum. Stet clita kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet. Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam voluptua. At vero eos et accusam et justo duo dolores et ea rebum. Stet clita kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet.",
                "watchers": [
                    {
                        "name": "John Cena"
                    },
                    {
                        "name": "John Cena"
                    },
                    {
                        "name": "John Cena"
                    },
                    {
                        "name": "John Cena"
                    },
                    {
                        "name": "John Cena"
                    }
                ]
            }
        ]
    }
}

Benchmark python strawberry

https://strawberry.rocks/

Server `Date` is missing from crystal benchmarks

This is a small changes, however an important one for benchmarking. A date header needs to be specified for each response. This is as per the RFC https://datatracker.ietf.org/doc/html/rfc2616#section-14.18

I have raised #233 to make sure that crystal does follow the spec. Feel free to find a better way to do this if you think there is one.

Checked with the other webservers, including tailcall, and found that they all send back a date header