coveooss / spillway

A simple, distributed and flexible rate limiter

Spillway

A distributed throttling solution

Spillway is an easy to use solution to add distributed throttling at the software level in your public API. This is particularly useful if multiple services are running in different JVMs. It is also possible to quickly to react when throttling happens with our built-in call-back mechanism.

Storage backend currently supported:

• In memory (for usage within the same JVM)
• Redis

All external storage can be (and should be) wrapped in our asynchronous storage to avoid slowing down/stopping queries if external problems occurs with the external storage.

Getting Started

Add Spillway to your project pom

<dependency>
    <groupId>com.coveo</groupId>
    <artifactId>spillway</artifactId>
    <version>2.0.0</version>
</dependency>

Documentation

The java documentation is available here: https://coveooss.github.io/spillway/

Usage

Sample 1

    LimitUsageStorage storage = new AsyncLimitUsageStorage(new RedisStorage("localhost"));
    SpillwayFactory spillwayFactory = new SpillwayFactory(storage);

    Limit<String> myLimit = LimitBuilder.of("myLimit").to(2).per(Duration.ofMinutes(1)).build();
    Spillway<String> spillway = spillwayFactory.enforce("myResource", myLimit);
    
    spillway.call("myLimit"); // nothing happens
    spillway.call("myLimit"); // nothing happens
    spillway.call("myLimit"); // throws SpillwayLimitExceededException

Sample 2

    LimitUsageStorage storage = new InMemoryStorage();
    SpillwayFactory spillwayFactory = new SpillwayFactory(storage);

    Limit<User> userLimit = LimitBuilder.of("perUser", User::getName).to(3).per(Duration.ofHours(1)).build();
    Limit<User> ipLimit = LimitBuilder.of("perIp", User::getIp).to(3).per(Duration.ofHours(1)).withExceededCallback(myCallback).build();
    Spillway<User> spillway = spillwayFactory.enforce("myResource", userLimit, ipLimit);

    User john = new User("john", "127.0.0.1");
    User gina = new User("gina", "127.0.0.1");

    spillway.tryCall(john); // true
    spillway.tryCall(gina); // true
    spillway.tryCall(john); // true
    spillway.tryCall(gina); // false, perIp limit exceeded.

Sample 3

    LimitUsageStorage storage = new InMemoryStorage();
    SpillwayFactory spillwayFactory = new SpillwayFactory(storage);
    
    LimitOverride override = LimitOverrideBuilder.of("john").to(10).per(Duration.ofHours(1)).build();
    Limit<String> userLimit = LimitBuilder.of("perUser").to(30).per(Duration.ofHours(1)).withLimitOverride(override).build();
    Spillway<User> spillway = spillwayFactory.enforce("myResource", userLimit);

    spillway.tryCall("john", 11); // false
    spillway.tryCall("gina", 20); // true

External Resources

cirrus-up-cloud wrote a nice blog post about using Spillway on AWS with Elasticache.

spillway's Issues

Refactor RedisStorage

• Remove all @Deprecated calls
• Super overloaded constructors should all be by builders

Triggers for calls that exceed the limit

Now that the limit is verified before incrementing, the triggers are still called. I'm not sure if this is the desired behaviour.

example:
Limit of 5, ValueTrigger of 3

• Storage = 0/5
• tryCall(6) = false
• Storage = 0/5
• Trigger executed
• tryCall(3) = true
• Storage = 2/5
• Trigger NOT executed

Other database

Since the project now has a maintainer, I would propose to add other database support if you have time.
Personally, I am closely watching ScyllaDB (http://www.scylladb.com/) which is a distributed NoSQL database like Cassandra, but way faster.

Change formatting

I suggest to change the formatting to the one we use in Cloud V2...

• tab = 4 spaces
• no strange line return after (
• no strange line return after operator
• etc

Refactor LimitUsageStorage

• Rename
• Add a get method

Have different way to enforce "rate limiting" vs "quotas"

This is all up for discussion but I'm dropping the idea here so that we can, um, discuss it :)

This library can be used for both limiting the rate of requests based on certain attributes (for protection against misbehaving callers), and also to enforce quotas allocated to certain resources (think customers, individual users, etc.). Those are very similar, but rate limiting often is done on a per-second basis, whereas quotas can use much larger intervals like hours or days. I think we could benefit by using a different implementation for those.

In particular, for rate-limiting we might be able to do a proper job without having to actively share data between nodes (using Redis for example). If we can somehow have a mostly up to date value for the number of servers that are actively processing requests (this might prove tricky), we could use an in-memory storage that only remembers the metrics for the current second. Each node would only allow MAXQPS/NODECOUNT queries per second.

Then, since we'd be using shared storage only for quotas, we could space out the intervals where we push data to it, reducing the load on that central server.

Refactor Spillway

• Have a tryCall that does not increment the counter useless all limits are not triggered

Callback can sometime be ignored when using distributed storage

If we are using the callback mechanism, the threshold can sometime be ignored if the sync increases the capacity over the threshold since we determine that the threshold was just reached using: current capacity - cost and verifying that it is under the limit.

Refactor LimitKey

We miss some information in the LimitKey, I had to use hacks to rebuild the AddAndGetRequest in the AsyncBatchLimitUsageStorage.

• Add enough information to have a valid AddAndGetRequest
• Refactor sync to use information

AsyncLimitUsageStorage memory leak in high concurrency situation

the call method 'AsyncLimitUsageStorage.addAndGet' occur memory leak, this will create too manay tasks in high concurrency situation, but there is only one thread that executes task to send data to distribute storage, so more and more tasks will be kept in long stock.

Error when executing keysCanExpire test

I have this error when I try to test the lib:
Failed tests:
RedisStorageTest.keysCanExpire:97 Not true that <2> is equal to <1>

It seems that the garbage does not have enough time OR that we don't look into the right bucket.

Support incrementing the call counter by a number that isn't 1

Sometimes you might want to limit a user to a certain number of bytes downloaded per day.
To do this, you'd need the rate-limiting to not be always incremented by 1.

ASync Batch Storage Optimization

As suggested here:
#19 (comment)

We should use a pipeline in the batch sync.

Allow batching of backend operations

Allow backend to receive multiple "addAndGet" so we can save on network and other computations.

Predicate/Property extractor for Limit Override

As proposed here: #17 (comment)
It would be nice to be able to specify a predicate or another property extractor for a Limit Override.
My main concern is how we should deal with multiple predicate/property extractor that match the current query.
We also want to be efficient, for that Property Extractor is better since we don't allow the user to specify its own verification, but its also less flexible.

Any news?

Any news about the project?
Does it behave well?
Did you found bugs?

Support dynamic configuration of limits.

Users will want their limits to be easily modified. You don't want them to be hardcoded at all times!

Support for standard libraries such as Archaius and perhaps an API + Simple UI would be very helpful.

Documentation for AWS integration

Hi,

I wrote a post describing how to integrate Spillway with AWS ElastiCache.
If you find useful, please link it in the library documentation.
Link to the article: https://tinyurl.com/ydxtnv7k

Split project into multiple modules

We don't want our Redis dependencies to be part of the core module.
We should split the project into different modules for different technologies.

Redis, Memcached, Spring, Servlet,...

Spring integration

It would be great if adding rate limiting to a spring application could be done via simple annotations.
Perhaps something could be done via autoproxy?

Add Support for Servlet Filters that do this automatically

It would be great if we could have a simple servlet filter that would allow us to easily add basic throttling to our applications

Implement proper cleanup in in-memory storage

Fix formatting

Formatting is very weird and not Cloud V2 standard

InMemmoryStorage optimization

We use synchronize in the InMemoryStorage, but we already use a ConcurrentHashMap. If we modify slightly the way we override the keys, it would possible to remove them.

Also, the ConcurrentHashMap uses the default values at creation (meaning its optimized for 16 threads), we should adjust that for our own needs (I suggest 4 threads to be more realist). The gain in performance should be significant.

Allow asynchronous backend operations

Use a in-memory variant, batch increment periodically to the backend.

Applications should have the choice, if they don't need the ultra-precise counter, they should be able to wrap the storage in a "good enough" estimate that doesn't negatively impact performance on every call to call.

Publish to Maven Central

Hi guys!

Do you think Spillway is ready to publish to Maven Central? If so, could you please do it?

Allow Limits to have a Warning callback

Warning is called when it passes a configurable threshold.

Should be generic enough. Perhaps refactor current limitExceededCallback mechanism to reuse that logic.