graphql-crystal / benchmarks Goto Github PK

GraphQL server benchmarks

License: MIT License

Dockerfile 1.16% Crystal 14.05% JavaScript 11.22% Rust 7.23% Go 5.32% C# 0.70% Python 3.36% Scala 3.25% Nim 3.26% Ruby 3.85% Elixir 39.09% Nix 2.16% D 5.33%

benchmarks's Issues

GraphQL Yoga results

Hi!
This is Arda from @the-guild-org. I am one of maintainers of GraphQL Yoga, and we recently introduces a lot of performance improvements on GraphQL Yoga. I'd like to kindly ask if you can update the results based on the latest version of GraphQL Yoga's node package.

Thanks!

What server specs were used in this benchmark?

Thanks for benching alot of the gql tools, however its unclear what server was used to perform these benchmarks. I am looking to do my own benchmarking as a comparison. Thanks

Warmup servers prior to executing benchmarks

Hi there 👋 I'm looking to add another Scala GraphQL library to the benchmarks. When I first looked at the current results I was a bit surprised that Sangria (Scala) was performing this badly. After looking at the benchmarking script, it seems that the servers are not warmed up prior to being benchmarked.

The JVM has a notoriously bad cold-start performance (see related post here), so any GraphQL libraries from JVM languages (Java, Kotlin, Scala, etc.) being benchmarked will show as having very poor performance if the server is not warmed up prior to measuring request latencies.

A very simple way to warmup the servers is to execute this line twice, discarding the results from the first invocation. Ideally, the first invocation should be with a smaller number of connections to prevent overwhelming the server as it warms up. I believe that this might also help resolve #217.

I'm happy to create a PR if you agree with this approach

Benchmarks don't run

/nix/store/j2gmfdd29cp3rcvghmn1a5kgcnmf8myp-binutils-2.40/bin/ld: cannot find -lpcre2-8 (this usually means you need to install the development package for libpcre2-8): No such file or directory

Dependency Dashboard

This issue provides visibility into Renovate updates and their statuses. Learn more

Open

These updates have all been created already. Click a checkbox below to force a retry/rebase of any.

Update dependency org.sangria-graphql:sangria to v3

Check this box to trigger a request for Renovate to run again on this repository

Benchmark Elixir / Absinthe

It would be great if it could be added to the comparison.

Thank you so much for the benchmark! 🙌

Use codespaces for Benchmarks

Currently the benchmarks run on GitHub actions which doesn't guarantee consistent cpu across runs. Its better to integrate it witg codespaces or setup a dedicated ci machine with consistent cpu.

Benchmark python strawberry

https://strawberry.rocks/

Carry over to reality

Hi!

First of all thanks a lot for these benchmarks 🤗

When running my own with very small payloads - like say{ "data": { "user": { "name": "John Cena" } } } - I can see a 7.5x difference between graphql-js and async-graphql, which checks out pretty well with the benchmarks provided here.

However when I go ahead and create larger payloads (attached below) by creating new objects/structs like one would in a real world scenario (where they'd come from an ORM or whatever), the difference melts down to about 3.5x - which is still considerable, but largely different to the 7.5x (small) or even your 10x (your benchmark, even smaller) payloads. Do you think it might make sense to make the example payload larger to be more like what you'd see in production (at least partially, for lists), or am I overlooking something? Could it be that with bigger payloads more connections must be kept alive and streaming takes longer, so the advantages of faster execution are diminished? Something isn't quite clicking for me.

Traffic is about 86 MBps (for the bigger payload) which is under the 4 Gbps or 500 MBps network out the Linode Dedicated 4GB can do by quite a margin, so network shouldn't be the bottleneck. Bombardier is run on a separate Linode Shared 16GB so that shouldn't bottleneck either. Tests where run using both internet and VLAN.

Edit: In my benchmarks, the small response was 144 Bytes and the large one 11.3 KB, so ~80x difference.

Larger Payload

{
    "data": {
        "todos": [
            {
                "title": "Lorem ipsum dolor sit amet",
                "description": "Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam voluptua. At vero eos et accusam et justo duo dolores et ea rebum. Stet clita kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet. Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam voluptua. At vero eos et accusam et justo duo dolores et ea rebum. Stet clita kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet.",
                "watchers": [
                    {
                        "name": "John Cena"
                    },
                    {
                        "name": "John Cena"
                    },
                    {
                        "name": "John Cena"
                    },
                    {
                        "name": "John Cena"
                    },
                    {
                        "name": "John Cena"
                    }
                ]
            },
            {
                "title": "Lorem ipsum dolor sit amet",
                "description": "Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam voluptua. At vero eos et accusam et justo duo dolores et ea rebum. Stet clita kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet. Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam voluptua. At vero eos et accusam et justo duo dolores et ea rebum. Stet clita kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet.",
                "watchers": [
                    {
                        "name": "John Cena"
                    },
                    {
                        "name": "John Cena"
                    },
                    {
                        "name": "John Cena"
                    },
                    {
                        "name": "John Cena"
                    },
                    {
                        "name": "John Cena"
                    }
                ]
            },
            {
                "title": "Lorem ipsum dolor sit amet",
                "description": "Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam voluptua. At vero eos et accusam et justo duo dolores et ea rebum. Stet clita kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet. Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam voluptua. At vero eos et accusam et justo duo dolores et ea rebum. Stet clita kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet.",
                "watchers": [
                    {
                        "name": "John Cena"
                    },
                    {
                        "name": "John Cena"
                    },
                    {
                        "name": "John Cena"
                    },
                    {
                        "name": "John Cena"
                    },
                    {
                        "name": "John Cena"
                    }
                ]
            },
            {
                "title": "Lorem ipsum dolor sit amet",
                "description": "Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam voluptua. At vero eos et accusam et justo duo dolores et ea rebum. Stet clita kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet. Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam voluptua. At vero eos et accusam et justo duo dolores et ea rebum. Stet clita kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet.",
                "watchers": [
                    {
                        "name": "John Cena"
                    },
                    {
                        "name": "John Cena"
                    },
                    {
                        "name": "John Cena"
                    },
                    {
                        "name": "John Cena"
                    },
                    {
                        "name": "John Cena"
                    }
                ]
            },
            {
                "title": "Lorem ipsum dolor sit amet",
                "description": "Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam voluptua. At vero eos et accusam et justo duo dolores et ea rebum. Stet clita kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet. Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam voluptua. At vero eos et accusam et justo duo dolores et ea rebum. Stet clita kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet.",
                "watchers": [
                    {
                        "name": "John Cena"
                    },
                    {
                        "name": "John Cena"
                    },
                    {
                        "name": "John Cena"
                    },
                    {
                        "name": "John Cena"
                    },
                    {
                        "name": "John Cena"
                    }
                ]
            },
            {
                "title": "Lorem ipsum dolor sit amet",
                "description": "Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam voluptua. At vero eos et accusam et justo duo dolores et ea rebum. Stet clita kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet. Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam voluptua. At vero eos et accusam et justo duo dolores et ea rebum. Stet clita kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet.",
                "watchers": [
                    {
                        "name": "John Cena"
                    },
                    {
                        "name": "John Cena"
                    },
                    {
                        "name": "John Cena"
                    },
                    {
                        "name": "John Cena"
                    },
                    {
                        "name": "John Cena"
                    }
                ]
            },
            {
                "title": "Lorem ipsum dolor sit amet",
                "description": "Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam voluptua. At vero eos et accusam et justo duo dolores et ea rebum. Stet clita kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet. Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam voluptua. At vero eos et accusam et justo duo dolores et ea rebum. Stet clita kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet.",
                "watchers": [
                    {
                        "name": "John Cena"
                    },
                    {
                        "name": "John Cena"
                    },
                    {
                        "name": "John Cena"
                    },
                    {
                        "name": "John Cena"
                    },
                    {
                        "name": "John Cena"
                    }
                ]
            },
            {
                "title": "Lorem ipsum dolor sit amet",
                "description": "Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam voluptua. At vero eos et accusam et justo duo dolores et ea rebum. Stet clita kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet. Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam voluptua. At vero eos et accusam et justo duo dolores et ea rebum. Stet clita kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet.",
                "watchers": [
                    {
                        "name": "John Cena"
                    },
                    {
                        "name": "John Cena"
                    },
                    {
                        "name": "John Cena"
                    },
                    {
                        "name": "John Cena"
                    },
                    {
                        "name": "John Cena"
                    }
                ]
            },
            {
                "title": "Lorem ipsum dolor sit amet",
                "description": "Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam voluptua. At vero eos et accusam et justo duo dolores et ea rebum. Stet clita kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet. Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam voluptua. At vero eos et accusam et justo duo dolores et ea rebum. Stet clita kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet.",
                "watchers": [
                    {
                        "name": "John Cena"
                    },
                    {
                        "name": "John Cena"
                    },
                    {
                        "name": "John Cena"
                    },
                    {
                        "name": "John Cena"
                    },
                    {
                        "name": "John Cena"
                    }
                ]
            },
            {
                "title": "Lorem ipsum dolor sit amet",
                "description": "Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam voluptua. At vero eos et accusam et justo duo dolores et ea rebum. Stet clita kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet. Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam voluptua. At vero eos et accusam et justo duo dolores et ea rebum. Stet clita kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet.",
                "watchers": [
                    {
                        "name": "John Cena"
                    },
                    {
                        "name": "John Cena"
                    },
                    {
                        "name": "John Cena"
                    },
                    {
                        "name": "John Cena"
                    },
                    {
                        "name": "John Cena"
                    }
                ]
            },
            {
                "title": "Lorem ipsum dolor sit amet",
                "description": "Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam voluptua. At vero eos et accusam et justo duo dolores et ea rebum. Stet clita kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet. Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam voluptua. At vero eos et accusam et justo duo dolores et ea rebum. Stet clita kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet.",
                "watchers": [
                    {
                        "name": "John Cena"
                    },
                    {
                        "name": "John Cena"
                    },
                    {
                        "name": "John Cena"
                    },
                    {
                        "name": "John Cena"
                    },
                    {
                        "name": "John Cena"
                    }
                ]
            },
            {
                "title": "Lorem ipsum dolor sit amet",
                "description": "Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam voluptua. At vero eos et accusam et justo duo dolores et ea rebum. Stet clita kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet. Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam voluptua. At vero eos et accusam et justo duo dolores et ea rebum. Stet clita kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet.",
                "watchers": [
                    {
                        "name": "John Cena"
                    },
                    {
                        "name": "John Cena"
                    },
                    {
                        "name": "John Cena"
                    },
                    {
                        "name": "John Cena"
                    },
                    {
                        "name": "John Cena"
                    }
                ]
            },
            {
                "title": "Lorem ipsum dolor sit amet",
                "description": "Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam voluptua. At vero eos et accusam et justo duo dolores et ea rebum. Stet clita kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet. Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam voluptua. At vero eos et accusam et justo duo dolores et ea rebum. Stet clita kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet.",
                "watchers": [
                    {
                        "name": "John Cena"
                    },
                    {
                        "name": "John Cena"
                    },
                    {
                        "name": "John Cena"
                    },
                    {
                        "name": "John Cena"
                    },
                    {
                        "name": "John Cena"
                    }
                ]
            },
            {
                "title": "Lorem ipsum dolor sit amet",
                "description": "Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam voluptua. At vero eos et accusam et justo duo dolores et ea rebum. Stet clita kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet. Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam voluptua. At vero eos et accusam et justo duo dolores et ea rebum. Stet clita kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet.",
                "watchers": [
                    {
                        "name": "John Cena"
                    },
                    {
                        "name": "John Cena"
                    },
                    {
                        "name": "John Cena"
                    },
                    {
                        "name": "John Cena"
                    },
                    {
                        "name": "John Cena"
                    }
                ]
            },
            {
                "title": "Lorem ipsum dolor sit amet",
                "description": "Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam voluptua. At vero eos et accusam et justo duo dolores et ea rebum. Stet clita kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet. Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam voluptua. At vero eos et accusam et justo duo dolores et ea rebum. Stet clita kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet.",
                "watchers": [
                    {
                        "name": "John Cena"
                    },
                    {
                        "name": "John Cena"
                    },
                    {
                        "name": "John Cena"
                    },
                    {
                        "name": "John Cena"
                    },
                    {
                        "name": "John Cena"
                    }
                ]
            }
        ]
    }
}

Benchmark for Sangria stuck

While raising the PR for new benchmark observing Sangria benchmark are getting stuck and eventually the Run action fail after 6Hr.

https://github.com/graphql-crystal/benchmarks/actions/runs/6730873449/job/18294413019

Server `Date` is missing from crystal benchmarks

This is a small changes, however an important one for benchmarking. A date header needs to be specified for each response. This is as per the RFC https://datatracker.ietf.org/doc/html/rfc2616#section-14.18

I have raised #233 to make sure that crystal does follow the spec. Feel free to find a better way to do this if you think there is one.

Checked with the other webservers, including tailcall, and found that they all send back a date header

OCaml and Haskell examples

Will be interesting to see those two implementations.

For Haskell probably https://morpheusgraphql.com/

And for OCaml they have integration with Dream framework: https://ceramichacker.com/blog/29-3x-building-graphql-apis-with-dream

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.

Jobs

Jooble