GithubHelp home page GithubHelp logo

express-rate-limit / rate-limit-redis Goto Github PK

View Code? Open in Web Editor NEW
183.0 4.0 33.0 1.04 MB

A rate limiting store for express-rate-limit with Redis/Redict/Valkey/etc.

Home Page: https://www.npmjs.com/package/rate-limit-redis

License: MIT License

Shell 0.41% TypeScript 99.59%
express express-js express-middleware expressjs node node-js nodejs rate-limit rate-limiter rate-limiting

rate-limit-redis's Introduction

rate-limit-redis

Github Workflow Status npm version GitHub Stars npm downloads

A redis store for the express-rate-limit middleware. Also supports redict & valkey.

Installation

From the npm registry:

# Using npm
> npm install rate-limit-redis
# Using yarn or pnpm
> yarn/pnpm add rate-limit-redis

From Github Releases:

# Using npm
> npm install https://github.com/express-rate-limit/rate-limit-redis/releases/download/v{version}/rate-limit-redis.tgz
# Using yarn or pnpm
> yarn/pnpm add https://github.com/express-rate-limit/rate-limit-redis/releases/download/v{version}/rate-limit-redis.tgz

Replace {version} with the version of the package that you want to use, e.g.: 3.0.0.

Usage

Importing

This library is provided in ESM as well as CJS forms, and works with both Javascript and Typescript projects.

This package requires you to use Node 16 or above.

Import it in a CommonJS project (type: commonjs or no type field in package.json) as follows:

const { RedisStore } = require('rate-limit-redis')

Import it in a ESM project (type: module in package.json) as follows:

import { RedisStore } from 'rate-limit-redis'

Examples

To use it with a node-redis client:

import { rateLimit } from 'express-rate-limit'
import { RedisStore } from 'rate-limit-redis'
import { createClient } from 'redis'

// Create a `node-redis` client
const client = createClient({
	// ... (see https://github.com/redis/node-redis/blob/master/docs/client-configuration.md)
})
// Then connect to the Redis server
await client.connect()

// Create and use the rate limiter
const limiter = rateLimit({
	// Rate limiter configuration
	windowMs: 15 * 60 * 1000, // 15 minutes
	max: 100, // Limit each IP to 100 requests per `window` (here, per 15 minutes)
	standardHeaders: true, // Return rate limit info in the `RateLimit-*` headers
	legacyHeaders: false, // Disable the `X-RateLimit-*` headers

	// Redis store configuration
	store: new RedisStore({
		sendCommand: (...args: string[]) => client.sendCommand(args),
	}),
})
app.use(limiter)

To use it with a ioredis client:

import { rateLimit } from 'express-rate-limit'
import { RedisStore } from 'rate-limit-redis'
import RedisClient from 'ioredis'

// Create a `ioredis` client
const client = new RedisClient()
// ... (see https://github.com/luin/ioredis#connect-to-redis)

// Create and use the rate limiter
const limiter = rateLimit({
	// Rate limiter configuration
	windowMs: 15 * 60 * 1000, // 15 minutes
	max: 100, // Limit each IP to 100 requests per `window` (here, per 15 minutes)
	standardHeaders: true, // Return rate limit info in the `RateLimit-*` headers
	legacyHeaders: false, // Disable the `X-RateLimit-*` headers

	// Redis store configuration
	store: new RedisStore({
		sendCommand: (command: string, ...args: string[]) =>
			client.send_command(command, ...args),
	}),
})
app.use(limiter)

Configuration

sendCommand

The function used to send commands to Redis. The function signature is as follows:

;(...args: string[]) => Promise<number> | number

The raw command sending function varies from library to library; some are given below:

Library Function
node-redis async (...args: string[]) => client.sendCommand(args)
ioredis async (command: string, ...args: string[]) => client.send_command(command, ...args)
handy-redis async (...args: string[]) => client.nodeRedis.sendCommand(args)
tedis async (...args: string[]) => client.command(...args)
redis-fast-driver async (...args: string[]) => client.rawCallAsync(args)
yoredis async (...args: string[]) => (await client.callMany([args]))[0]
noderis async (...args: string[]) => client.callRedis(...args)

prefix

The text to prepend to the key in Redict/Redis.

Defaults to rl:.

resetExpiryOnChange

Whether to reset the expiry for a particular key whenever its hit count changes.

Defaults to false.

License

MIT © Wyatt Johnson, Nathan Friedly, Vedant K

rate-limit-redis's People

Contributors

amitgurbani avatar dependabot[bot] avatar gamemaker1 avatar gp2mv3 avatar jotto avatar knoxcard avatar knoxcard2 avatar maxime-guyot avatar mifi avatar mindtraveller avatar misowei-noodoe avatar nfriedly avatar rishabhrao avatar vincentcr avatar wyattjoh avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar

rate-limit-redis's Issues

client.sendCommand is not a function

Description

var RateLimit = require("express-rate-limit");
var RateLimitRedis = require("rate-limit-redis");
const config = require("config");
const redis = require("redis");
var redisStoreConnection = {
host: config.redis.host,
port: config.redis.port,
};
let client = redis.createClient(redisStoreConnection).connect();

module.exports.limited100 = RateLimit({
store: new RateLimitRedis({
sendCommand: (...args) => client.sendCommand(args),
}),
max: 100, // start blocking after 50 requests
message: "30.50: Too Many Requests Have been Made wait awhile and try again",
});

When i compile this i get the following error:

client.sendCommand is not a function

I use the limited100 as a decorator in some of my endpoint calls in node.

Library version

3.0.1

Node version

16.15

Typescript version (if you are using it)

No response

Module system

CommonJS

Redis Client

redis

Invalid node version in documentation

Description

image image

In node v14 there is a problem due to replaceAll

Library version

4.1.2

Node version

v14

Typescript version (if you are using it)

No response

Module system

ESM

Redis instance size

Hi, we're interested in using this software for our server but I'm not sure how bug the redis instance should be. Do you maybe know how much memory is used for every request / ip?

Thanks

expiry param make no sense

Since expiry time is already set in the Rate Limit, why should we include it another time for the redis?

I guess that if it's not possible due to the express-rate-limit module, we can suggest modification of the implementation for the store driver on the express-rate-limit

Cannot install rate-limit-redis 3.1.0 with express-rate-limit 7.0.0

Description

I've installed latest version (7.0.0) of express-rate-limit and wanted to add Redis store. However, when I tried to install latest rate-limit-redis I got following error:

npm ERR! Found: [email protected]
npm ERR! node_modules/express-rate-limit
npm ERR!   express-rate-limit@"^7.0.0" from the root project
npm ERR! 
npm ERR! Could not resolve dependency:
npm ERR! peer express-rate-limit@"^6" from [email protected]
npm ERR! node_modules/rate-limit-redis
npm ERR!   rate-limit-redis@"*" from the root project
npm ERR! 
npm ERR! Fix the upstream dependency conflict, or retry
npm ERR! this command with --force or --legacy-peer-deps
npm ERR! to accept an incorrect (and potentially broken) dependency resolution.

Library version

3.1.0

Node version

v18.16.0

Typescript version (if you are using it)

No response

Module system

CommonJS

Publish the new version

I would like to use the new ioredis functionality however the package doesn't appear to be published with the code from #2

If you could publish the new version that would be great!

Outdated npm

Hi, it seems like published npm version is not your latest release version. Eg. passIfNotConnected option is missing.

Unable to use rate-limit-redis.

Description

code below is not working as expected

const express = require("express");
const morgan = require("morgan");

const app = express();

// enable json parser
app.use(express.json());

// enable http logging
app.use(morgan("dev"));

// add redis using tedis
const { Tedis } = require("tedis");

const client = new Tedis({
  port: 6379,
  host: "127.0.0.1",
});

// add redis limiter
const rateLimit = require("express-rate-limit");
const RedisStore = require("rate-limit-redis");

const limiter = rateLimit({
  windowMs: 15 * 60 * 1000, // 15 minutes
  max: 10, // Limit each IP to 10 requests per `window` (here, per 15 minutes)
  standardHeaders: true, // Return rate limit info in the `RateLimit-*` headers
  legacyHeaders: false, // Disable the `X-RateLimit-*` headers,
  message: "Too many requests from this IP, please try again in 15 minutes.",
  // Redis store configuration
  store: new RedisStore({
    sendCommand: (...args) => client.command(...args),
    // sendCommand: async (...args) => client.command(...args),
    // sendCommand: async (...args) => await client.command(...args),
    // ☝️ commented ones are also not working
  }),
});

app.use(limiter);

// health check endpoint
app.get("/health", (_, res) => {
  res.send({
    status: "UP",
    message: "Everything is fine",
  });
});

// initialize the app
const PORT = 3000;
app.listen(PORT, () => {
  console.log(`Server is running on port ${PORT}`);
});

Code's Current Behavior (in order)

  1. when a GET req made on curl -X GET localhost:3000/health
  2. server stuck and do not response
  3. inside Redis it creates key rl:::1
  4. when another GET req made curl -X GET localhost:3000/health
  5. it prompts error TypeError: Expected result to be array of values at RedisStore.increment
  6. inside Redis rl:::1 gets incremented

Error

TypeError: Expected result to be array of values
    at RedisStore.increment (/home/mrsha2nk/Codex/learn-rate-limiter/node_modules/rate-limit-redis/dist/index.cjs:69:13)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async /home/mrsha2nk/Codex/learn-rate-limiter/node_modules/express-rate-limit/dist/index.cjs:158:38
    at async /home/mrsha2nk/Codex/learn-rate-limiter/node_modules/express-rate-limit/dist/index.cjs:141:5

Information

  • using Redis container docker run -d -p 6379:6379 --name redis redis
  • node version: v18.4.0
  • npm version: 8.12.1
  • express-rate-limit: 6.5.1
  • rate-limit-redis: 3.0.1
  • tedis: 0.1.12

Library version

3.0.1

Node version

v18.4.0

Typescript version (if you are using it)

No response

Module system

CommonJS

Redis Client

tedis

Fallback store?

Hello! First of all, thanks for making this module.

Is there a way to use a secondary store (like in-memory) when for whatever reason redis has become unreachable?

Connect with Unix Socket?

How do I connect to a Unix Socket?

When the server and client benchmark programs run on the same box, both the TCP/IP loopback and unix domain sockets can be used. Depending on the platform, unix domain sockets can achieve around 50% more throughput than the TCP/IP loopback (on Linux for instance). The default behavior of redis-benchmark is to use the TCP/IP loopback.
The performance benefit of unix domain sockets compared to TCP/IP loopback tends to decrease when pipelining is heavily used (i.e. long pipelines).

https://redis.io/topics/benchmarks

Intermittently receiving: TypeError: Cannot read property 'map' of null

Hey there,

We are occasionally seeing TypeError: Cannot read property 'map' of null in our logs.

The culprit seems to be this function when the replies object passed to it is null:

var processReplies = function(replies) { // <--- replies is null
    // in ioredis, every reply consists of an array [err, value].
    // We don't need the error here, and if we aren't dealing with an array,
    // nothing is changed.
    return replies.map(function(val) { // <--- therefore this line blows up 
      if (Array.isArray(val) && val.length >= 2) {
        return val[1];
      }
      return val;
    });
  };

However, I can't figure out why this function would be called as it seems that the error is being handled correctly here:

    options.client.multi()
      .incr(rdskey)
      .pttl(rdskey)
      .exec(function(err, replies) {
        if (err) {
          return cb(err);
        }
       //...

Now, the simple fix is to handle replies object being null. However, there seems to be a deeper issue here since I can't imagine in which scenario both err and replies can be null?

If it helps, we are using IORedis and have HA mode enabled.

Any ideas?

RedisStore is required with each limiter

Description

This not an bug, but just an issue I faced, which will hopefully save somebody hours of work.

For most this is probably already obvious, but you need to create a new RedisStore for each limiter.

You cannot define a RedisStore and then re-use it accross all limiters. This does NOT work:

const defaultSettings = {
    standardHeaders: true,
    legacyHeaders: false,
    store: new RedisStore({
            sendCommand: (...args: string[]) => client.sendCommand(args),
            prefix: 'rateLimit:',
    }),
};

const limiterOne = rateLimit({
    ...defaultSettings,
    windowMs: 24 * 60 * 60 * 1000, // 24 hours
    max: 500, 
});


const limiterTwo = rateLimit({
    ...defaultSettings,
    windowMs: 60 * 1000, // 1 min
    max: 10, 
});

It doesn't work, because RedisStore reads the windowMs property when initializing, and uses it to set reset-timer. Without it, the reset-timer is set by default to 24 hours. Additionally, in the above example the RedisStore would always use the same key prefix, which means that different API routes would count towards the same limit.

So in short, you always need to create a new RedisStore with each limiter.

const defaultSettings = {
    standardHeaders: true,
    legacyHeaders: false,
};

const limiterOne = rateLimit({
    ...defaultSettings,
    windowMs: 24 * 60 * 60 * 1000, // 24 hours
    max: 500, 
    store: new RedisStore({
            sendCommand: (...args: string[]) => client.sendCommand(args),
            prefix: 'rateLimitOne:',
    }),
});


const limiterTwo = rateLimit({
    ...defaultSettings,
    windowMs: 60 * 1000, // 1 min
    max: 10, 
    store: new RedisStore({
            sendCommand: (...args: string[]) => client.sendCommand(args),
            prefix: 'rateLimitTwo:',
    }),

EXTRA

You can extract the sendCommand function to re-use it across all new RedisStores.

const defaultSettings = {
    standardHeaders: true,
    legacyHeaders: false,
};

const sendCommand = (...args) => client.sendCommand(args);

const limiterOne = rateLimit({
    ...defaultSettings,
    windowMs: 24 * 60 * 60 * 1000, // 24 hours
    max: 500, 
    store: new RedisStore({
            sendCommand,
            prefix: 'rateLimitOne:',
    }),
});


const limiterTwo = rateLimit({
    ...defaultSettings,
    windowMs: 60 * 1000, // 1 min
    max: 10, 
    store: new RedisStore({
            sendCommand,
            prefix: 'rateLimitTwo:',
    }),

'passIfNotConnected' option is not compatible with ioredis client

I love the idea of the passIfNotConnected option that was recently added in #29

However, it is based on the redis ("node-redis") client implementation of the client.connected Boolean, which is not compatible with the ioredis client.

In ioredis, they seem to use the following complex set of checks for determining if the client is connected (e.g. source example):

if (
  this.status === "connecting" ||
  this.status === "connect" ||
  this.status === "ready"
) {
  reject(new Error("Redis is already connecting/connected"));
  return;
}

Add types to exports in package.json

Description

As title, it should be great if we add types: ./dist/index.d.ts to exports in package.json.

Why

There are types at '.../node/node_modules/rate-limit-redis/dist/index.d.ts', but this result could not be resolved when respecting package.json "exports". The 'rate-limit-redis' library may need to update its package.json or typings.

I got this message while using RedisStore in TypeScript.

Alternatives

I don't know if there's another way to achieve this.

Now I create my own module declaration to avoid error.

incr is not a function

Unhandled Error TypeError: options.client.multi(...).incr is not a function
    at RedisStore.incr (/Users/user/Documents/GitHub/demo/node_modules/rate-limit-redis/lib/redis-store.js:46:8)
    at rateLimit (/Users/user/Documents/GitHub/demo/node_modules/express-rate-limit/lib/express-rate-limit.js:59:19)
    at Layer.handle [as handle_request] (/Users/user/Documents/GitHub/demo/node_modules/express/lib/router/layer.js:95:5)
    at trim_prefix (/Users/user/Documents/GitHub/demo/node_modules/express/lib/router/index.js:317:13)
    at /Users/user/Documents/GitHub/demo/node_modules/express/lib/router/index.js:284:7
    at Function.process_params (/Users/user/Documents/GitHub/demo/node_modules/express/lib/router/index.js:335:12)
    at next (/Users/user/Documents/GitHub/demo/node_modules/express/lib/router/index.js:275:10)
    at Layer.handle [as handle_request] (/Users/user/Documents/GitHub/demo/node_modules/express/lib/router/layer.js:91:12)
    at trim_prefix (/Users/user/Documents/GitHub/demo/node_modules/express/lib/router/index.js:317:13)
    at /Users/user/Documents/GitHub/demo/node_modules/express/lib/router/index.js:284:7
    at Function.process_params (/Users/user/Documents/GitHub/demo/node_modules/express/lib/router/index.js:335:12)
    at next (/Users/user/Documents/GitHub/demo/node_modules/express/lib/router/index.js:275:10)
    at /Users/user/Documents/GitHub/demo/node_modules/request-ip/dist/index.js:164:5
    at Layer.handle [as handle_request] (/Users/user/Documents/GitHub/demo/node_modules/express/lib/router/layer.js:95:5)
    at trim_prefix (/Users/user/Documents/GitHub/demo/node_modules/express/lib/router/index.js:317:13)
    at /Users/user/Documents/GitHub/demo/node_modules/express/lib/router/index.js:284:7
(node:23587) UnhandledPromiseRejectionWarning: TypeError: executor is not a function
    at /Users/user/Documents/GitHub/demo/server/redis.ts:77:9
    at new Promise (<anonymous>)
    at demoRedis.multi (/Users/user/Documents/GitHub/demo/server/redis.ts:74:12)
    at RedisStore.incr (/Users/user/Documents/GitHub/demo/node_modules/rate-limit-redis/lib/redis-store.js:45:20)
    at rateLimit (/Users/user/Documents/GitHub/demo/node_modules/express-rate-limit/lib/express-rate-limit.js:59:19)
    at Layer.handle [as handle_request] (/Users/user/Documents/GitHub/demo/node_modules/express/lib/router/layer.js:95:5)
    at trim_prefix (/Users/user/Documents/GitHub/demo/node_modules/express/lib/router/index.js:317:13)
    at /Users/user/Documents/GitHub/demo/node_modules/express/lib/router/index.js:284:7
    at Function.process_params (/Users/user/Documents/GitHub/demo/node_modules/express/lib/router/index.js:335:12)
    at next (/Users/user/Documents/GitHub/demo/node_modules/express/lib/router/index.js:275:10)
    at Layer.handle [as handle_request] (/Users/user/Documents/GitHub/demo/node_modules/express/lib/router/layer.js:91:12)
    at trim_prefix (/Users/user/Documents/GitHub/demo/node_modules/express/lib/router/index.js:317:13)
    at /Users/user/Documents/GitHub/demo/node_modules/express/lib/router/index.js:284:7
    at Function.process_params (/Users/user/Documents/GitHub/demo/node_modules/express/lib/router/index.js:335:12)
    at next (/Users/user/Documents/GitHub/demo/node_modules/express/lib/router/index.js:275:10)
    at /Users/user/Documents/GitHub/demo/node_modules/request-ip/dist/index.js:164:5

query: count in redis increase by 2 for everyhit

MGET rl:::ffff:127.0.0.1 => 2
MGET rl:::ffff:127.0.0.1 => 4
...etc

my configuration

const RateLimit = require('express-rate-limit');
const RedisStore = require('rate-limit-redis');
const limiter = new RateLimit({
  store: new RedisStore({
    expiry: process.env.RATE_LIMIT_WINDOW_IN_SEC,
    prefix: 'rl:',
    client: redisClient.getClient(),
  }),
  windowMs: 60000,
  max: process.env.RATE_LIMIT_ALLOWED_REQUESTS_PER_IP, // limit each IP to x requests per windowMs
  delayMs: 0, // disable delaying - full speed until the max limit is reached
  message: "Too many API hits from your IP, please try again after sometime",
});
//  apply to all requests
app.use(limiter);

Expired Key TTL is reset without value reset causing increment on stale key value

Description

Redis take some time to clean up the expired keys. During the time, if the increment call is made it increments the stale key value and resets the TTL on it.

To reproduce:
Using it with express-rate-limit and putting a load of just 1 RPS when rate limit window is 1 second.

Code incrementing the stale keys

const result = await this.sendCommand(
      'SCRIPT',
      'LOAD',
      `
          local totalHits = redis.call("INCR", KEYS[1])
          local timeToExpire = redis.call("PTTL", KEYS[1])
          if timeToExpire <= 0 or ARGV[1] == "1"
          then
            redis.call("PEXPIRE", KEYS[1], tonumber(ARGV[2]))
            timeToExpire = tonumber(ARGV[2])
          end

          return { totalHits, timeToExpire }
      `
        .replace(/^\s+/gm, '')
        .trim()
    );

Fix:

const result = await this.sendCommand(
      'SCRIPT',
      'LOAD',
      `
          local totalHits = redis.call("INCR", KEYS[1])
          local timeToExpire = redis.call("PTTL", KEYS[1])
          if timeToExpire <= 0 or ARGV[1] == "1"
          then
            if ARGV[1] ~= "1" then
              redis.call("SET", KEYS[1], "1")
              totalHits = 1
            end
            redis.call("PEXPIRE", KEYS[1], tonumber(ARGV[2]))
            timeToExpire = tonumber(ARGV[2])
          end

          return { totalHits, timeToExpire }
      `
        .replace(/^\s+/gm, '')
        .trim()
    );

Library version

3.0.1

Node version

16

Typescript version (if you are using it)

No response

Module system

CommonJS

Configure redis

Hey, thanks for your plugin.
I have some troubles to configure the plugin with my heroku redis

Here is my config

const store = new RedisStore({
	host: redisUrl.hostname, // loremipsum.amazon.com....
	password: redisUrl.auth, // h:dfkjsj27934jkdhfksdfgksjdhé
	port: redisUrl.port // 37259
})

Here is the error I get :

Redis connection to 127.0.0.1:6379 failed - connect ECONNREFUSED

Thank your for your help 👍

Unable to handle connection loss

Description

If my express server loses connection to my Redis server rate-limit-redis throws an exception which crashes my server.

I tried adding error handling to my sendCommand callback function but I was unable to find a solution that satisfies the library. After looking at the library code a bit I think the library will need to be updated to better handle errors.

It would be nice for the server not to crash and for the request to just be denied or allowed based on a fallback setting, when connection to Redis fails. In my case if Redis was down I would prefer to allow all request in temporally, but I can see why other would rather block all request.

Side note, it could be helpful if there was a an option in the future to fallback to an in memory store when Redis is down, this way the API can't be totally abused.

Library version

3.0.2

Node version

v16.13.0

Typescript version (if you are using it)

No response

Module system

CommonJS

Redis Client

redis 4.6.7

Store is not valid error

Stumped on this one.

Error: The store is not valid.
...node_modules/express-rate-limit/lib/express-rate-limit.js:37:11

I'm using the following packages:

"rate-limit-redis": "^1.5.0",
"redis": "^2.8.0",
"express-rate-limit": "^3.1.1",
var redis = require('redis');
var redisClient = redis.createClient(config.redis.url);

redisClient.on("error", function (err) {
    console.log("Error:", err);
});
redisClient.on("connect", function (msg) {
    console.log("Redis connection online");
});
var RateLimit = require("express-rate-limit");
var RedisRateLimit = require("rate-limit-redis");
app.enable("trust proxy");
var limiter = new RateLimit({
  windowMs: 10 * 60 * 1000, // 10 minutes
  max: 125, // limit each IP # of requests per windowMs
  delayMs: 0, // disable delaying - full speed until the max limit is reached
  store: new RedisStore({
    client: redisClient,
    expiry: 10*60*1000, 
    prefix: "rl"
  })
});
app.use(limiter);

ioredis setup

Description

The README example didn't work for ioredis.

This is what worked for me:

import Redis from "ioredis";

const redis = new Redis();
const client = redis.createClient();

const rateLimiter = rateLimit({
  windowMs: 15 * 60 * 1000, // 15 minutes interval
  max: 2000,
  standardHeaders: true,
  legacyHeaders: false,
  store: new RedisStore({
    sendCommand: (...args) => client.call(...args),
  }),
  delayMs: 0,
});

Library version

"ioredis": "^4.28.3", "rate-limit-redis": "^3.0.0",

Node version

v14.18.2

Typescript version (if you are using it)

No response

Module system

CommonJS

expiry clarification

Hi,

Can you explain to me the difference between
express-rate-limit windowMs and rate-limit-redis expiry?

windowMs: milliseconds - how long to keep records of requests in memory. Defaults to 60000 (1 minute).

expiry: seconds - how long each rate limiting window exists for. Defaults to 60.

Does one of these override the other? Or should they be set the same? Or do they do completely different things?

redis in docker-compose

The connection code for redis

const RateLimitStore = require('rate-limit-redis');
const redis = require('redis');
const redisClient = redis.createClient({
	port: process.env.REDIS_PORT || undefined,
	host: process.env.REDIS_HOST || undefined,
	password: process.env.REDIS_PASSWORD || undefined,
	db: process.env.REDIS_DB || undefined,
});
new RateLimitStore({ client: redisClient, resetExpiryOnChange: true, prefix: "create" }), windowMs: 60 * 60 * 1000, max: 4, message: "You are creating too many obj. Please try again later." }

My docker-compose

version: "3.7"
services:
  app:
    image: app
    container_name: app
    environment:
      - REDIS_HOST=redis_db
      - REDIS_PORT=6379
      # postgres
      - POSTGRES_DB_USERNAME=opentogethertube
      - POSTGRES_DB_NAME=opentogethertube
      - POSTGRES_DB_HOST=postgres_db
      - POSTGRES_DB_PASSWORD=postgres
    ports:
      - 8080:8080
    links:
      - redis_db
      - postgres_db
    depends_on:
      - redis_db
      - postgres_db
    restart: "${DOCKER_RESTART_POLICY:-unless-stopped}"

  redis_db:
    container_name: app_redis
    image: redis
    healthcheck:
      test: "redis-cli ping"
    expose:
      - 6379
    volumes:
      - db-data-redis:/data
    restart: "${DOCKER_RESTART_POLICY:-unless-stopped}"
2020-04-21 19:48:18 app info Server started on port 8080
events.js:292
throw er; // Unhandled 'error' event
      ^

Error: Redis connection to 127.0.0.1:6379 failed - connect ECONNREFUSED 127.0.0.1:6379
     at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1141:16)
 Emitted 'error' event on RedisClient instance at:
     at RedisClient.on_error (/usr/app/node_modules/rate-limit-redis/node_modules/redis/index.js:406:14)
     at Socket.<anonymous> (/usr/app/node_modules/rate-limit-redis/node_modules/redis/index.js:279:14)
     at Socket.emit (events.js:315:20)
     at emitErrorNT (internal/streams/destroy.js:84:8)
     at processTicksAndRejections (internal/process/task_queues.js:84:21) {
   errno: -111,
   code: 'ECONNREFUSED',
   syscall: 'connect',
   address: '127.0.0.1',
   port: 6379
 }

to enter the broken container

app:
...
command: tail -f /dev/null
...

if I change the default value of the host in the /node_modules/rate-limit-redis/node_modules/redis/index.js line 70
cnx_options.host = options.host || 'redis_db';
then run the app it work perfectly fine

store type error: RedisClient or Redis ?

"ioredis": "^4.14.1",
"redis": "^2.8.0",
"express-rate-limit": "^5.0.0",
"rate-limit-redis": "^1.7.0",

//in typescript
import IORedis = require('ioredis'); 
const redis = new IORedis({}) //type:Redis.Redis

import Redis = require('redis'); 
const redisClient = Redis.createClient({})// type:Redis.RedisClient

import RateLimit = require('express-rate-limit')
import RedisStore = require('rate-limit-redis')

const limiter = new RateLimit({
  store: new RedisStore({
    client: client  // need type:  RedisClient
  }),
  max: 100, 
  delayMs: 0 
});

So, how to solve it?

Error with @types/rate-limit-redis

@types/rate-limit-redis dependent 'StoreIncrementCallback' of express-rate-limit, but it not export module named StoreIncrementCallback in the latest version.
image

Get "NOSCRIPT No matching script. Please use EVAL" when redis pod restarts.

Description

Please consider adding a try catch around https://github.com/wyattjoh/rate-limit-redis/blob/effebd57bf943d3acfb3cdf3c4856c72383cd749/src/lib.ts#L114 and https://github.com/wyattjoh/rate-limit-redis/blob/effebd57bf943d3acfb3cdf3c4856c72383cd749/src/lib.ts#L121, so that when RedisNoScriptException happens, loadScript() can be invoked again to load the script into redis.

Current workaround:

const app = express()
class MyRedisStore extends RedisStore { 
    increment = async(key) => {
        let results = []
        try {
            results = await this.sendCommand(
                "EVALSHA",
                await this.loadedScriptSha1,
                "1",
                this.prefixKey(key),
                this.resetExpiryOnChange ? "1" : "0",
                this.windowMs.toString()
            );
        } catch (err) {
            this.loadedScriptSha1 = await this.loadScript()
            results = await this.sendCommand(
                "EVALSHA",
                await this.loadedScriptSha1,
                "1",
                this.prefixKey(key),
                this.resetExpiryOnChange ? "1" : "0",
                this.windowMs.toString()
            );
        }
      
        if (!Array.isArray(results)) {
            throw new TypeError("Expected result to be array of values");
        }
    
        if (results.length !== 2) {
            throw new Error(`Expected 2 replies, got ${results.length}`);
        }
    
        const totalHits = results[0];
        if (typeof totalHits !== "number") {
            throw new TypeError("Expected value to be a number");
        }
    
        const timeToExpire = results[1];
        if (typeof timeToExpire !== "number") {
            throw new TypeError("Expected value to be a number");
        }
    
        const resetTime = new Date(Date.now() + timeToExpire);
        return {
            totalHits,
            resetTime,
        };
    }
}

function apiLimiterDefault() {
    return rateLimit({
        ........
        store: new MyRedisStore({
            sendCommand: (...args) => xdrRedis.boot().handle().sendCommand(args),
        })
    })
}

app.get("/", [apiLimiterDefault()], (req, res) {
  return res.json({"hello": "world"})
})

(Basically just wrapping a try catch block around)

Library version

3.0.1

Node version

14.16.0

Typescript version (if you are using it)

No response

Module system

CommonJS

Redis Client

redis 4.0.4

document and recommend usage of send_command() instead of call() for ioredis

Description

In the README for ioredis usage, the function call() is used to implement sendCommand as the following. B/c of the non-present type for the call function, a @ts-expect-error is used.

// Redis store configuration
  store: new RedisStore({
    // @ts-expect-error - Known issue: the `call` function is not present in @types/ioredis
    sendCommand: (...args: string[]) => client.call(...args),
  }),

ioredis has another function send_command which seems to do just what you are looking for and is supported by @types/ioredis. The usage would look like the following

   sendCommand: (...args) => {
      const [command, ...commandArgs] = args;
      return client.send_command(command, ...commandArgs);
    },

Would you recommend the use of send_command() in the README?

Why

The call() function is not supported by @types/redis while send_command() is supported. It is much more intuitive to use.

Alternatives

The alternative is to continue using call() with the @ts-expect-error tag while unable to actually get first-class support in the IDE's.

evalsha error when rebooting dev redis

Description

when rebooting the redis instance on my dev environment, but leave the connection "alive", all requests afterwarsd result into:

ReplyError: NOSCRIPT No matching script. Please use EVAL.
at parseError (/app/node_modules/redis-parser/lib/parser.js:179:12)
at parseType (/app/node_modules/redis-parser/lib/parser.js:302:14) {
command: {
name: 'EVALSHA',
args: [
'3f424a504987d231baa2e19159bb5def8ad32f87',
'1',
'rl-testing:134.209.252.110:/app-api/',
'0',
'1000'
]

Library version

3.0.1

Node version

16.x

Typescript version (if you are using it)

16.x

Module system

CommonJS

Rate limiting should be ignored when Redis client is unreachable

Description

If the Redis client throws an error, such as ECONNREFUSED 127.0.0.1:6379, RedisStore should act as a useless middleware instead of blocking the API where it's being used. Catching the Redis client error does not help since RedisStore will still throw an error about the Redis client.

Why

Rate limiting is not vital for an API like a database and it should not prevent it from working in case Redis is unreachable.

Alternatives

Catching Redis client errors, catching inside sendCommand.

signalLimiter = rateLimit({
            windowMs: 1 * 60 * 1000,
            max: global.gConfig.MAX_REQUESTS_PER_MINUTE, 
            standardHeaders: true,
            legacyHeaders: false,
            store: new RedisStore({
                sendCommand: (...args) => {
                    try {
                        return redisClient.sendCommand(args);
                    }
                    catch (err) {
                        console.log("An error occurred while sending Redis command", err);
                        return true; // does not work
                    }
                }
            }),
        });

Desired behavior when Redis client error occurs:

signalLimiter ~= (req, res, next) => next();

PromiseRejectionHandledWarning: Promise rejection was handled asynchronously

Description

I get the following error the first time a request comes in:

(node:66707) PromiseRejectionHandledWarning: Promise rejection was handled asynchronously (rejection id: 4)
    at handledRejection (node:internal/process/promises:172:23)
    at promiseRejectHandler (node:internal/process/promises:118:7)
    at evalCommand (...)
    at RedisStore.runCommandWithRetry (...)
    at RedisStore.increment (...)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async node_modules/express-rate-limit/dist/index.cjs:553:5

Possibly because of a missing await here: https://github.com/express-rate-limit/rate-limit-redis/blob/f9b187878a554a89495c349e43a31af17abcb731/source/lib.ts#L110C14-L110C14

Library version

^6.10.0

Node version

v18.12.1

Typescript version (if you are using it)

5.0.4

Module system

CommonJS

Buffer deprecation warning

The current redis version, 2.8.0, causes Buffer deprecation warning messages:

[DEP0005] DeprecationWarning: Buffer() is deprecated due to security and usability issues. Please use the Buffer.alloc(), Buffer.allocUnsafe(), or Buffer.from() methods instead.
(it looks like one of redis dependencies, redis-parser 2.6.0, is the reason this happens).

I understand this was fixed in the latest version.

Is there any chance to to bump the redis version to avoid these warnings?

Unable to create rate limiter with redis store (io-redis)

Description

I have been able to use the rate limiter just fine with the memory store, however when I try to use the redis store (io-redis), I get the following error.

/usr/app/server/backend/src/config/rateLimiter.ts:40
  store: new RedisStore( {
         ^
TypeError: rate_limit_redis_1.default is not a constructor
    at Object.<anonymous> (/usr/app/server/backend/src/config/rateLimiter.ts:40:10)
    at Module._compile (node:internal/modules/cjs/loader:1256:14)
    at Module.m._compile (/usr/app/server/backend/node_modules/ts-node/src/index.ts:1618:23)
    at Module._extensions..js (node:internal/modules/cjs/loader:1310:10)
    at Object.require.extensions.<computed> [as .ts] (/usr/app/server/backend/node_modules/ts-node/src/index.ts:1621:12)
    at Module.load (node:internal/modules/cjs/loader:1119:32)
    at Function.Module._load (node:internal/modules/cjs/loader:960:12)
    at Module.require (node:internal/modules/cjs/loader:1143:19)
    at require (node:internal/modules/cjs/helpers:110:18)
    at Object.<anonymous> (/usr/app/server/backend/src/app.ts:25:1)

Here is what I'm trying to do:

import rateLimit from 'express-rate-limit';
import RedisStore from 'rate-limit-redis';
import RedisClient from 'ioredis';

// Create a `ioredis` client
const client = new RedisClient( {
  host: <connection string>,
  port: <number>,
  db: <number>
} );

export const rateLimiter = rateLimit( {
  windowMs: 10 * 1000, // 10 seconds
  limit: 100, // Limit each IP to 100 requests per `window` (here, per 10 seconds).
  standardHeaders: 'draft-7', // draft-6: `RateLimit-*` headers; draft-7: combined `RateLimit` header
  legacyHeaders: false, // Disable the `X-RateLimit-*` headers.

  // Redis store configuration
  store: new RedisStore( {
    // @ts-expect-error - Known issue: the `call` function is not present in @types/ioredis
    sendCommand: async ( ...args: string[] ): unknown => client.call( ...args ),
  } ),
} );

Library version

7.1.2

Node version

18.16.1

Typescript version (if you are using it)

4.9.5

Module system

ESM

Make "redis" an optional dependency

I use rate-limit-redis with ioredis as my preferred Redis client library.

While inspecting my projects dependencies I noticed that rate-limit-redis has a hard dependency on the redis NPM package. From the documentation, rate-limit-redis is compatible with either ioredis or redis

I suggest making redis an optional dependency. This would allow consumers of rate-limit-redis to choose which Redis client(s) are installed.

Higher response time

Description

Hello!

I am using this module with ioredis. Without this module my response time is around 30-50 ms, but with this module, my response time is around 100 - 150 ms. I am hosting a free redis instance on redislabs. Is this normal?

On the moment, this isn't really an issue. But will this block me if i scale to millions of requests / second?

Library version

v3.0.1

Node version

v16.13.0

Typescript version (if you are using it)

No response

Module system

CommonJS

rate-limit-redis works with `redis` and not with `ioredis`

We're migrating from the redis module to ioredis.
Once migrated the rate-limit-redis module stops working and generates an error message:
redis client is not connected, rate-limit-redis disabled!
Despite the fact the client is actually connected.

The following code works:

const Redis = require('redis')
const redisClient = Redis.createClient({
  host: env.vars.REDIS_HOST,
  port: env.vars.REDIS_PORT,
  db: env.vars.REDIS_DB,
  password: env.vars.REDIS_SECRET_KEY,
  tls: env.vars.REDIS_TLS,
  connect_timeout: env.vars.REDIS_TIMEOUT,
  prefix: env.vars.REDIS_KEYS_PREFIX
})

While this one doesn't work:

const IoRedis = require('ioredis')
const redisClient = new IoRedis({
  host: env.vars.REDIS_HOST,
  port: env.vars.REDIS_PORT,
  db: env.vars.REDIS_DB,
  password: env.vars.REDIS_SECRET_KEY,
  tls: env.vars.REDIS_TLS,
  connectTimeout: env.vars.REDIS_TIMEOUT,
  keyPrefix: env.vars.REDIS_KEYS_PREFIX
})

Once the redis client is created, it is used as a new RedisStore:

const rateLimiter = rateLimit({
    ...
    store: new RedisStore({
      expiry: 60,
      resetExpiryOnChange: false,
      prefix: `ratelimiter:tier${tierIndex + 1}:`,
      client: redisClient,
      passIfNotConnected: true // If Redis is not connected, let the request succeed as failover
    })
  })
})

Multiple instances in same redis

I use multiple limiters with diferent configuration in my backend. Do you think could i have problems using the same instance of redis?

hi, in my local pc I can't have my ip

what I see in redis is: "rl:::ffff:127.0.0.1" but not IP. is this the behavior in development ?

rateLimit({
windowMs: windowMs || 60 * 60 * 1000,
max: max || 10,
handler: handler || expressRateLimitMiddleware,
store: new RateLimitRedisStore({
client: redisClient,
expiry: 10601000,
prefix: "rl"
}),
})

Issue with @nestjs/graphql

Description

I've installed latest version (7.0.2) of express-rate-limit and wanted to add Redis store. However, when I tried to run app when installed latest rate-limit-redis (4.1.0) I got following error when running my nest backend:

[0] Error: Cannot find module '@nestjs/graphql'
[0] Require stack:
[0] - /app_name/backend/node_modules/ncsrf/dist/guards/csrf.guard.js
[0] - /app_name/backend/node_modules/ncsrf/dist/decorators/csrf.decorator.js
[0] - /app_name/backend/node_modules/ncsrf/dist/decorators/index.js
[0] - /app_name/backend/node_modules/ncsrf/dist/index.js
[0] - /app_name/backend/src/main.ts
[0]     at Function.Module._resolveFilename (node:internal/modules/cjs/loader:933:15)
[0]     at Function.Module._resolveFilename.sharedData.moduleResolveFilenameHook.installedValue (/app_name/backend/node_modules/@cspotcode/source-map-support/source-map-support.js:811:30)
[0]     at Function.Module._resolveFilename (/app_name/backend/node_modules/tsconfig-paths/src/register.ts:115:36)
[0]     at Function.Module._load (node:internal/modules/cjs/loader:778:27)
[0]     at Module.require (node:internal/modules/cjs/loader:1005:19)
[0]     at require (node:internal/modules/cjs/helpers:102:18)
[0]     at Object.<anonymous> (/app_name/backend/node_modules/ncsrf/dist/guards/csrf.guard.js:14:19)
[0]     at Module._compile (node:internal/modules/cjs/loader:1101:14)
[0]     at Module._extensions..js (node:internal/modules/cjs/loader:1153:10)
[0]     at Object.require.extensions.<computed> [as .js] (/app_name/backend/node_modules/ts-node/src/index.ts:1608:43) {
[0]   code: 'MODULE_NOT_FOUND',
[0]   requireStack: [
[0]     '/app_name/backend/node_modules/ncsrf/dist/guards/csrf.guard.js',
[0]     '/app_name/backend/node_modules/ncsrf/dist/decorators/csrf.decorator.js',
[0]     '/app_name/backend/node_modules/ncsrf/dist/decorators/index.js',
[0]     '/app_name/backend/node_modules/ncsrf/dist/index.js',
[0]     '/app_name/backend/src/main.ts'
[0]   ]
[0] }

Library version

4.1.0

Node version

16.13.0

Typescript version (if you are using it)

4.7.4

Module system

ESM

ECONNREFUSED when initializing

I initialize the redis client connection and then pass it on to the rate limiter, as a new redis store, and I'm getting

Error: Redis connection to 127.0.0.1:6379 failed - connect ECONNREFUSED 127.0.0.1:6379
     at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1134:16)

Here's my code:

import * as redis from 'redis';
import * as RateLimiter from 'express-rate-limit';
import * as RedisStore from 'rate-limit-redis';

const redisClient = redis.createClient({
    host: config.host,
    port: config.port,
    password: config.password
  });

new RateLimiter({
    store: new RedisStore({
      client: redisClient
    }),
    windowMs,
    max: maxReq
});

If I comment out the following part of the code:

    store: new RedisStore({
      client: redisClient
    }),

then everything initializes fine. And actually I've been using this Redis connection for a few weeks now, before trying to add the rate limiter.

Unable to work with redis when cluster mode is enabled

Description

We are using redis in cluster mode. When I test it with redis with cluster mode enabled, sendCommand functions hangs and never returns any response.

import rateLimit from "express-rate-limit";
import { createCluster } from "redis";
import RedisStore from "rate-limit-redis";

const createRateLimiter = async(({}) => {
  const cluster = createCluster({
    rootNodes: [
      {
        url: "redis://127.0.0.1:6379",
      },
    ],
  });
  await cluster.connect();

  return rateLimit({
    store: new RedisStore({
      sendCommand: (...args) => cluster.sendCommand(args),
    }),
  });
});

Library version

3.0.0

Node version

v14.16.1

Typescript version (if you are using it)

No response

Module system

CommonJS

RedisStore throws error when client object is passed

const client = createClient();
const requestLimiter = rateLimit({
  store: new RedisStore({
    client
  }),
  windowMs: 1 * 60 * 1000 * 60, // 1h
  max: 100, // Limit IP to 100 requests per hour
  message: "Request limit exceeded! try again in a short while",
});

Throws the following error:

TypeError: options.client.multi(...).incr(...).pttl is not a function

Any solution please?
I need to use the redis container in my docker network

Expiry time is inconsistent with express-rate-limit

The express-rate-limit package uses windowMs option to set the interval when limit should reset. In redis terms windowMs is supposed to set the expiry timeout. However now this is controlled by a standalone expiry option in rate-limit-redis, so windowMs is basically completely ignored. So code like this:

rateLimit({
  max: 100,
  windowMs: 1000,
  store: new RedisStore({...})
})

is supposed to allow 100 requests per second, but since the default option for expiry is 60 seconds, in reality this allows 100 per minute which is complete nonsense. It's also not clear how this could be fixed, because it would probably also need some changes in express-rate-limit package which is problematic. At least it could be written in documentation that one should omit windowMs option and use expiry instead when instantiating rateLimit middleware

Inconsistent documentation

The docs don't over the example properties in the readme:

const limiter = new RateLimit({
  store: new RedisStore({
    client: client
  }),
  max: 100, // limit each IP to 100 requests per windowMs
  delayMs: 0 // disable delaying - full speed until the max limit is reached
});

Only expiry, resetExpiryOnChange, prefix, client and redisURL are listed, and these are presumably for the store configuration property. This makes the documentation very confusing imo. store, max and delayMs aren't documented, and windowMs (which only appears in a comment) isn't listed either.

Would you consider a PR to fix this?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.