hyper63 / hyper-adapter-redis

Redis adapter for Hyper Cache port

hyper

Use Clean Cloud Architecture to Scale on your Terms

Docs · Issues · What's New

• Introduction
• Status
• Running Locally
• Documentation
• Contributions
• Developer Setup
• Thank you

Introduction

👋 Hey 👋

Welcome to the hyper Service Framework project ⚡️!

Hyper is a multi-cloud, multi-language, service framework that enables best-of-breed cloud service utilization, from any cloud provider. By providing a simple and extensible context-bound API for the Cloud, the hyper Service Framework decouples application services from the infrastructure that powers them, positioning application software for growth and long term stability.

As a result, the hyper Service Framework helps software organizations build "optimally-scaled" software and software teams.

Hyper organizations tame technical debt, by using Clean Cloud Architecture to sensibly scale their software and software teams, only when the complexity is needed, and not all up-front.

Status

• Development
• Alpha
• Beta
• v1.0

Running Locally

You can run a hyper server locally, with 5 locally running hyper services, using hyper-nano, a precompiled executable binary of the hyper service framework.

curl https://hyperland.s3.amazonaws.com/hyper -o hyper-nano
chmod +x hyper-nano
./hyper-nano

Alternatively, if you're using node, you can run npx hyper-nano

This command will run a hyper server on PORT 6363 and store data for each hyper service in a directory named __hyper__ placed in the cwd [Ctrl/Cmd] - C will stop the service.

This nano version of hyper implements the following ports and adapters:

• data (powered by In-Memory MongoDB)
• cache (powered by Sqlite)
• storage (powered by your local file system)
• search (powered by Sqlite and Minisearch)
• queue (powered by Sqlite and an in-memory queue)

Learn more about hyper-nano and read our blog post

Documentation

For more information about hyper63 go to our documentation site. https://docs.hyper.io if you are unable to find the information you are looking for, post a question in our slack

Contributions

See Contributing to hyper.

Developer Setup

Fork this repo and use gitpod!

Gitpod

Thank you

• OpenSource Community
• CharlestonJS Community
• JRS Coding School Team
• And everyone else that has helped this project become successful!

hyper-adapter-redis's Issues

feat: set `COUNT` on `SCAN` command used on `listDocs`

listDocs is implemented using a scan against Redis. scan accepts a count argument:

Basically with COUNT the user specified the amount of work that should be done at every call in order to retrieve elements from the collection. This is just a hint for the implementation, however generally speaking this is what you could expect most of the times from the implementation.

The count defaults to 10, which is a really small number for most Redis caches. For example to iterate over a small database with just 1000 keys, scan with count 10 would result in 100 consecutive calls to scan. Redis has the ability to iterate keys quickly . From their docs:

Redis running on an entry level laptop can scan a 1 million key database in 40 milliseconds.

but the consecutive network calls, from scan, coupled with Redis' single-threaded runtime, can result in blocking and poor performance across concurrent scan calls, similar to the drawbacks of keys.

To mitigate this, we can empirically set the COUNT argument to something larger, say a static number like a 1000. Or a more sophisticated option would be to take into account the current number of keys in the database and choose a COUNT that guarantees a max or approximate max number of iterations by scan to iterate over all of the keys.

return hyper-esque errors

update cloud.hyper adapters to return hyper-esque errors on managed errors and rejected promises on an unhandled exception.

Each adapter method in the case of handled exceptions should return a resolved promise that resolves to a hyper error

A hyper error consists of:

ok: false
msg: optional: string
status: optional: integer
Unhandled exceptions the adapter should return a rejected promise. We should get this oob.

bug(cluster-mode): CROSSSLOT Keys in request don't hash to the same slot

In the redis adapter, the destroyStore method currently aggregates all keys prefixed with the name of the cache service, then attempts to delete all of those keys using del command. del is a multi-key command.

The CROSSSLOT error is thrown by Redis, when an operation is performed against multiple keys, and they keys do not belong to the same hash slot, basically not on the same node in the cluster.

feat: use hash tags to force all keys for a store into the same hash slot

Related to #16

As part of #16, each key is deleted sequentially, when deleting a hyper cache store. This works, but isn't the most efficient operation.

See #16 (comment)

Perhaps long term, we use hash tags for the benefits listed above.

Each key stored in Redis has the hyper store name prepended to the key ie. ${store)-${key}. If we hash tag the store name portion of the key ie {${store}}-${key}, Redis will place all of the keys for that particular hyper cache store onto the same node in the cluster, because all the keys in that hyper cache storage will map to the same hash slot.

https://redis.io/docs/reference/cluster-spec/#hash-tags

chore - add pre-commit hook to repo

cache API end to end testing

• create cache
• remove cache
• query cache documents

---

• create cache document
• get cache document
• update cache document
• remove cache document

bug: negative ttl causes redis error to be returned

Instead, a negative ttl should effectively immediately expire the document

upgrade redis for cluster support

query on sufficiently large keyspace causes "maximum callstack exceeded"

This is due to the implementation being recursive. So can just refactor to use a trampoline or other iterative approach.

bug: TypeError: options.client.clusterMeet is not a function

bug: redis-cluster support

the current adapter does not work correctly for a Redis cluster, the Redis driver as of version 23 has support for Redis clusters, but the documentation is not exactly clear. What we need to do is create a local Redis cluster and run the prescribed commands to make sure that the Redis driver is working correctly.

bug: `scan` on `listDocs` is only performed against a single node, in `clustered` Redis

We use scan in listDocs. This works as is with a Non-clustered Redis, but with Clustered Redis, scan must be performed against each node in the cluster, thus requiring a map-reduce pattern to gather all matches, across nodes in the cluster.

See https://www.dontpanicblog.co.uk/2022/07/30/scan-a-redis-cluster/ for potential approaches.

Another approach is to use what is described in #17 to force all keys in a store to the same slot ie. the same node. Since all keys are prefixed with the hyper cache store name, we can hashtag the store name, forcing all keys in a store to the same hash slot.

Ergo, If we hashtag the store name, a scan on a single node will work in both clustered and non-clustered Redis.