Define a base to accept a framework about benchmarks

[dougwilson]

Right, it is starting to get out of hand. For example is this:
a) a module to show how fast Fastify is for people who are curious about Fastify, or
b) a generic Node.js http framework benchmarker.

If a, then it makes sense to me that the limit would be the frameworks that are being used by the target audience Fastify wants to switch to Fastify, right?

[StarpTech]

In my opinion we should keep it very simple. Only compare frameworks which provide at least the featureset:

• Router
• Middleware mechanism
• Hook mechanism (optional)
  = the features which has the most impact on the rps.

• a minimum count of downloads per week.

But we should be open for innovations which don't comply with the download rate.

[nwoltman]

The number of frameworks in the benchmark has grown very large. I agree that it would be a good idea to have some criteria that can be used to determine which frameworks should and shouldn't be part of the benchmark. Here's my suggestion:

1. It should support something like middleware or hooks
2. It should include a router
   1. The router should support named parameters (e.g. /user/:id) and the matched parameters should be available in handlers/middleware (e.g. req.params.id)
3. It should make sending a JSON response easier than writing:

   res.setHeader('content-type', 'application/json')
   res.end(JSON.stringify({ hello: 'world' }))

   e.g. With Fastify, that can be written as: reply.send({ hello: 'world' })
   With Koa: ctx.body = { hello: 'world' }
   With Hapi: return { hello: 'world' }

[nwoltman]

Hmm, "first-class JSON" might be a little too specific. I think the main thing I'm trying to get at with point 3 is that the framework should do more than just pass Node's req/res through a router and middleware.

For example, Fastify has a bunch of great features like decorators, lifecycle hooks, and async handlers. Express extends req/res with many properties and functions that make handling requests easier. Koa has a unique request/response flow based on async/await. All of these frameworks are more than just a router + middleware. Perhaps "makes sending a JSON response easier" is just one of the ways we can tell if a framework is more than a simple router + middleware.

[mcollina]

I think the effort to implement and maintain the list is too high.
I would prefer a minimum set of features that each framework (or combination) has to provide for the benchmarks to be meaningful.

However, in the fastify README we should limit them for popularity/downloads.

Do we need a router? Does it needs to have middleware/plugins?

[delvedor]

I would prefer a minimum set of features that each framework (or combination) has to provide for the benchmarks to be meaningful.

I don't think this would work, because every framework has its own features. Maybe we could add a feature list table, like we did in: https://github.com/delvedor/router-benchmark#router-features
In this way every user can choose the framework knowing more than just the req/sec number.

In any case I think we should not accept frameworks with few downloads per week.

[allevo]

From my point of view, citing the router framework (aka request/sec) is not enough to choose the http framework.
Probably we need to filter the list keeping the framework (or framework + library) that implements some comparable features.

We need to know the maturity or maintenance of the protect also

[aichholzer]

I am with @allevo and @StarpTech on this one.
For instance, there is this PR which has 23 stars but seems relatively stable.

On the other hand, I believe everyone has the right to be benchmarked (and listed) (personally, I am not a fan of exclusion). That is, of course, provided we want to become something like a "benchmarking aggregator", the likes of this, for instance.

To start -maybe- stick to Nodejs frameworks?

[delvedor]

But we should be open for innovations which don't comply with the download rate.

The question is, how do you define if a project is innovative or not?

In my opinion have dozens of framework in our benchmark is useless and confusing for the reader, there are other projects for that.
The aim of this repository is to compare the most used frameworks in the Node.js community (and help us to see where Fastify places itself).

[jsumners]

If a, then it makes sense to me that the limit would be the frameworks that are being used by the target audience Fastify wants to switch to Fastify, right?

Yes.

[aichholzer]

@delvedor -hence my comment:

That is, of course, provided we want to become something like a "benchmarking aggregator", the likes of this, for instance.

@dougwilson -agreed. Now the question is which ones to keep as part of the benchmark?

[Eomm]

It should make sending a JSON response easier than writing:

Could we say that JSON should be the "first-class" value?

Then we can create a PR template of it