echovault / echovault

Embeddable and distributed in-memory alternative to Redis.

---

Table of Contents

1. What is SugarDB
2. Features
3. Usage (Embedded)
4. Usage (Client-Server)
   1. Homebrew
   2. Docker
   3. GitHub Container Registry
   4. Binaries
5. Clients
6. Benchmarks
7. Commands
   1. ACL
   2. ADMIN
   3. CONNECTION
   4. GENERIC
   5. HASH
   6. LIST
   7. PUBSUB
   8. SET
   9. SORTED SET
   10. STRING

What is SugarDB?

SugarDB is a highly configurable, distributed, in-memory data store and cache implemented in Go. It can be imported as a Go library or run as an independent service.

SugarDB aims to provide a rich set of data structures and functions for manipulating data in memory. These data structures include, but are not limited to: Lists, Sets, Sorted Sets, Hashes, and much more to come soon.

SugarDB provides a persistence layer for increased reliability. Both Append-Only files and snapshots can be used to persist data in the disk for recovery in case of unexpected shutdowns.

Replication is a core feature of SugarDB and is implemented using the RAFT algorithm, allowing you to create a fault-tolerant cluster of SugarDB nodes to improve reliability. If you do not need a replication cluster, you can always run SugarDB in standalone mode and have a fully capable single node.

SugarDB aims to not only be a server but to be importable to existing projects to enhance them with SugarDB features, this capability is always being worked on and improved.

Features

Features offered by SugarDB include:

1. TLS and mTLS support for multiple server and client RootCAs.
2. Replication cluster support using the RAFT algorithm.
3. ACL Layer for user Authentication and Authorization.
4. Distributed Pub/Sub functionality.
5. Sets, Sorted Sets, Hashes, Lists and more.
6. Persistence layer with Snapshots and Append-Only files.
7. Key Eviction Policies.
8. Command extension via shared object files.
9. Command extension via embedded API.
10. Multi-database support for key namespacing.

We are working hard to add more features to SugarDB to make it much more powerful. Features in the roadmap include:

1. Sharding
2. Streams
3. Transactions
4. Bitmap
5. HyperLogLog
6. Lua Modules
7. JSON
8. Improved Observability

Usage (Embedded)

Install SugarDB with: go get github.com/echovault/sugardb.

Here's an example of using SugarDB as an embedded library. You can access all of SugarDB's commands using an ergonomic API.

func main() {
  server, err := sugardb.NewSugarDB()
  if err != nil {
    log.Fatal(err)
  }

  _, _, _ = server.Set("key", "Hello, SugarDB!", sugardb.SETOptions{})

  v, _ := server.Get("key")
  fmt.Println(v) // Hello, SugarDB!

  // (Optional): Listen for TCP connections on this SugarDB instance.
  server.Start()
}

An embedded SugarDB instance can still be part of a cluster, and the changes triggered from the API will be consistent across the cluster.

Usage (Client-Server)

Homebrew

To install via homebrew, run:

1. brew tap echovault/sugardb
2. brew install echovault/echovault/sugardb

Once installed, you can run the server with the following command: sugardb --bind-addr=localhost --data-dir="path/to/persistence/directory"

Docker

docker pull echovault/sugardb

The full list of tags can be found here.

Container Registry

docker pull ghcr.io/echovault/sugardb

The full list of tags can be found here.

Binaries

You can download the binaries by clicking on a release tag and downloading the binary for your system.

Clients

SugarDB uses RESP, which makes it compatible with existing Redis clients.

Benchmarks

The following benchmark only applies to the TCP client-server mode.

Hardware: MacBook Pro 14in, M1 chip, 16GB RAM, 8 Cores
Command: redis-benchmark -h localhost -p 7480 -q -t ping,set,get,incr,lpush,rpush,lpop,rpop,sadd,hset,zpopmin,lrange,mset
Result:

PING_INLINE: 89285.71 requests per second, p50=0.247 msec                   
PING_MBULK: 85543.20 requests per second, p50=0.239 msec                   
SET: 65573.77 requests per second, p50=0.455 msec                   
GET: 79176.56 requests per second, p50=0.295 msec                   
INCR: 68870.52 requests per second, p50=0.439 msec                   
LPUSH: 27601.44 requests per second, p50=1.567 msec                   
RPUSH: 61842.92 requests per second, p50=0.519 msec                   
LPOP: 58548.01 requests per second, p50=0.567 msec                   
RPOP: 68681.32 requests per second, p50=0.439 msec                   
SADD: 67613.25 requests per second, p50=0.479 msec                   
HSET: 56561.09 requests per second, p50=0.599 msec                   
ZPOPMIN: 70972.32 requests per second, p50=0.359 msec                   
LPUSH (needed to benchmark LRANGE): 26434.05 requests per second, p50=1.623 msec                   
LRANGE_100 (first 100 elements): 26939.66 requests per second, p50=1.263 msec                   
LRANGE_300 (first 300 elements): 5081.82 requests per second, p50=9.095 msec                    
LRANGE_500 (first 500 elements): 2554.87 requests per second, p50=18.191 msec                   
LRANGE_600 (first 600 elements): 1903.96 requests per second, p50=24.607 msec                   
MSET (10 keys): 56022.41 requests per second, p50=0.463 msec 

Supported Commands

ACL

• ACL CAT
• ACL DELUSER
• ACL GETUSER
• ACL LIST
• ACL LOAD
• ACL SAVE
• ACL SETUSER
• ACL USERS
• ACL WHOAMI

ADMIN

• COMMAND COUNT
• COMMAND LIST
• COMMANDS
• LASTSAVE
• MODULE LIST
• MODULE LOAD
• MODULE UNLOAD
• REWRITEAOF
• SAVE

CONNECTION

• AUTH
• HELLO
• PING
• SELECT
• SWAPDB

GENERIC

• DECR
• DECRBY
• DEL
• EXPIRE
• EXPIRETIME
• FLUSHALL
• FLUSHDB
• GET
• INCR
• INCRBY
• MGET
• MSET
• PERSIST
• PEXPIRE
• PEXPIRETIME
• PTTL
• RENAME
• SET
• TTL

HASH

• HDEL
• HEXISTS
• HGET
• HGETALL
• HINCRBY
• HINCRBYFLOAT
• HKEYS
• HLEN
• HRANDFIELD
• HSET
• HSETNX
• HSTRLEN
• HVALS

LIST

• LINDEX
• LLEN
• LMOVE
• LPOP
• LPUSH
• LPUSHX
• LRANGE
• LREM
• LSET
• LTRIM
• RPOP
• RPUSH
• RPUSHX

PUBSUB

• PSUBSCRIBE
• PUBLISH
• PUBSUB CHANNELS
• PUBSUB NUMPAT
• PUBSUB NUMSUB
• PUNSUBSCRIBE
• SUBSCRIBE
• UNSUBSCRIBE

SET

• SADD
• SCARD
• SDIFF
• SDIFFSTORE
• SINTER
• SINTERCARD
• SINTERSTORE
• SISMEMBER
• SMEMBERS
• SMISMEMBER
• SMOVE
• SPOP
• SRANDMEMBER
• SREM
• SUNION
• SUNIONSTORE

SORTED SET

• ZADD
• ZCARD
• ZCOUNT
• ZDIFF
• ZDIFFSTORE
• ZINCRBY
• ZINTER
• ZINTERSTORE
• ZLEXCOUNT
• ZMPOP
• ZMSCORE
• ZPOPMAX
• ZPOPMIN
• ZRANDMEMBER
• ZRANGE
• ZRANGESTORE
• ZRANK
• ZREM
• ZREMRANGEBYLEX
• ZREMRANGEBYRANK
• ZREMRANGEBYSCORE
• ZREVRANK
• ZSCORE
• ZUNION
• ZUNIONSTORE

STRING

• GETRANGE
• SETRANGE
• STRLEN
• SUBSTR

echovault's Issues

Can provide more methods for `config`?

Since the config package is an internal package and cannot be referenced, I want to configure fields such as DataDir by calling echovault.NewEchoVault(). Can I add more custom configurations for echovault.NewEchoVault()?

Create eviction policies

In order to keep memory usage in control, it's essential to have eviction policies that allow us to purge keys. As part of this issue, we would like to have the following eviction policies implemented:

1. noeviction - Do not evict any keys regardless of memory usage.
2. allkeys-lru - Use an LRU algorithm to remove the least recently used keys.
3. allkeys-lfy - Use an LFU algorithm to remove the least frequently used keys.
4. volatile-lru - Use an LRU algorithm to remove the least recently used keys with the expire property set to true.
5. volatile-lfu - Use an LFU algorithm to remove the least frequently used keys with the expire property set to true.
6. allkeys-random - Randomly remove any of the keys.
7. volatile-random - Randomly remove any of the keys that have the expire property set to true.
8. volatile-ttl - Remove keys with the shortest remaining ttl.

Create test suite for ACL module

Create test cases that cover all the commands in the ACL module.

Encrypt AOF & Snapshot Files in Standalone Mode

Use a stream cypher to encrypt the AOF and Snapshot files in standalone mode. The user will have to configure an encryption key.

Create docusaurus docs for echovault website

Generate docusaurus website for echovault's web home.

Implement INCRBYFLOAT command

Implement the INCRBYFLOAT command, which increments the value at the specified key by the specified float amount if the value is an integer or a float.
Reference: https://redis.io/docs/latest/commands/incrbyfloat/

Client-Server Spec:

Command File: ./internal/modules/generic/commands.go
Test File: ./internal/modules/generic/commands_test.go

Command: incrbyfloat
Module: constants.GenericModule
Categories: constants.WriteCategory, constants.FastCategory
Description: (INCRBYFLOAT key decrement) Increment the string representing a floating point number stored at key by the specified increment. By using a negative increment value, the result is that the value stored at the key is decremented (by the obvious properties of addition). If the key does not exist, it is set to 0 before performing the operation.
Sync: true

Embedded Spec:

Command File: ./echovault/api_generic.go
Test File: ./echovault/api_generic_test.go

Implement HELLO command

Implement HELLO command that modifies the connection's RESP version and optionally authenticates the connection.
Reference: https://redis.io/docs/latest/commands/hello/

Client-Server Spec:

Command File: ./internal/modules/connection/commands.go
Test File: ./internal/modules/connection/commands_test.go

Command: hello
Module: constants.ConnectionModule
Categories: constants.ConnectionCategory, constants.FastCategory
Description: (HELLO [protover [AUTH username password] [SETNAME clientname]]) Switch to a different protocol, optionally authenticating and setting the connection's name, or provide a contextual client report.
Sync: false

Server requires root

I tried running the server in the cmd/ directory, but got this error: 2024/06/15 22:47:03 new preamble store -> mkdir error: mkdir /var/lib/echovault: permission denied

Maybe the /var/lib/echovault dir should default to ~/.local/lib/echovault when not running as root?

Implement DECR command

Implement the DECR command, which decrements the value at the specified key by 1 if the value is an integer or a float.
Reference: https://redis.io/docs/latest/commands/decr/

Client-Server Spec:

Command File: ./internal/modules/generic/commands.go
Test File: ./internal/modules/generic/commands_test.go

Command: decr
Module: constants.GenericModule
Categories: constants.WriteCategory, constants.FastCategory
Description: (DECR key) Decrements the number stored at key by one. If the key does not exist, it is set to 0 before performing the operation. An error is returned if the key contains a value of the wrong type or contains a string that can not be represented as integer.
Sync: true

Embedded Spec:

Command File: ./echovault/api_generic.go
Test File: ./echovault/api_generic_test.go

Implement RENAME command

Implement the RENAME command, which renames the specified key with the new name.
Reference: https://redis.io/docs/latest/commands/rename/

Client-Server Spec:

Command File: ./internal/modules/generic/commands.go
Test File: ./internal/modules/generic/commands_test.go

Command: rename
Module: constants.GenericModule
Categories: constants.KeyspaceCategory, constants.WriteCategory, constants.SlowCategory
Description: (RENAME key newkey) Renames the specified key with the new provided name.
Sync: true

Embedded Spec:

Command File: ./echovault/api_generic.go
Test File: ./echovault/api_generic_test.go

NOTE: You may have to edit the HandlerFuncParams type to include a function that renames keys. Alternatively, you could use the GetValues, DeleteKeys and SetValues methods together to delete the old key and set a new key with the old value. The first approach would be more atomic, though.

At the moment, only replication clusters exist without sharding. So the key should be renamed across the entire cluster.

Compatibility with Redis Clients

Improve compatibility with Redis clients to allow EchoVault to be a potential drop-in replacement for Redis. The following considerations need to be made:

1. Handshake with Redis client.
2. Make sure each command returns a valid RESP response.
3. Make Pub/Sub module compatible with Redis client.

Refactor command modules to use dependency injection of required parameters for command handlers

At the moment, HandlerFunc accepts a parameter that implements the EchoVault interafce. This causes the command modules to depend on the the echovault package, forcing the command modules to be placed in the pkg folder which is not idea.

The fix is to use Inversion of control to pass in all the functions, context, command and connection needed by the command handler, thus removing the dependence on the echovault package in any of the command modules.

This fix should also allow us to move the modules folder from /pkg to /internal. It should also make it easier to implement shared object file extension of echovault commands.

Implement SHUTDOWN command

The SHUTDOWN command shuts the server down synchronously.
Reference: https://redis.io/docs/latest/commands/shutdown/

Client-Server Spec:

Command File: ./internal/modules/admin/commands.go
Test File: ./internal/modules/admin/commands_test.go

Command: shutdown
Module: constants.AdminModule
Categories: contants.AdminCategory, constants.SlowCategory, constants.DangerousCategory
Description: (SHUTDOWN [NOSAVE | SAVE] [NOW] [FORCE] [ABORT]) synchronously shutdown the server
Sync: false

Embedded Spec:

Command File: ./echovault/api_admin.go
Test File: ./echovault/api_admin_test.go

Documentation

Add documentation to ./docs/docs/commands/admin/shutdown.mdx

Make EchoVault Embeddable

Make EchoVault embeddable so any application can enhance its feature set with EchoVault features by simply importing EchoVault.

Bonus: Allow projects to import some modules from EchoVault. For example, import only the Append-Only engine or the Snapshot engine.

Allow shared object (.so) plugins to be loaded into commands list

As a user, I should be able to pass a list of .so files to be loaded on startup in order to extend EchoVault functionality with custom commands.

The plugins should be available in both client-server and embedded mode. For embedded mode, provide a generic function that accepts a custom []string to trigger custom plugin commands. It should return an error or a []byte representing the RESP response from the command handler.

Implement GETEX command

Implement the GETEX command, which gets the current value at the specified key and then sets an expiry for the key.
Reference: https://redis.io/docs/latest/commands/getex/

Client-Server Spec:

Command File: ./internal/modules/generic/commands.go
Test File: ./internal/modules/generic/commands_test.go

Command: getex
Module: constants.GenericModule
Categories: constants.WriteCategory, constants.FastCategory
Description: (GETEX key [EX seconds | PX milliseconds | EXAT unix-time-seconds | PXAT unix-time-milliseconds | PERSIST]) Get the value of key and optionally set its expiration. GETEX is similar to [GET], but is a write command with additional options.
Sync: true

Embedded Spec:

Command File: ./echovault/api_generic.go
Test File: ./echovault/api_generic_test.go

Extend EchoVault functionality with Lua modules

Embedded Lua Engine that allows users to build Lua modules that can be used to extend commands supported by EchoVault.

Implement HMGET command

Implement the HMGET command, which gets the values of multiple fields from a hash.
Reference: https://redis.io/docs/latest/commands/hmget/

Client-Server Spec:

Command File: ./internal/modules/hash/commands.go
Test File: ./internal/modules/hash/commands_test.go

Command: hmget
Module: constants.HashModule
Categories: contants.HashCategory, constants.ReadCategory, constants.FastCategory
Description: (HMGET key field [field ...]) Returns the values associated with the specified fields in the hash stored at key.
Sync: false

Embedded Spec:

Command File: ./echovault/api_hash.go
Test File: ./echovault/api_hash_test.go

Implement TYPE command

Implement the TYPE command, which returns the data type of the provided key.
Reference: https://redis.io/docs/latest/commands/type/

Client-Server Spec:

Command File: ./internal/modules/generic/commands.go
Test File: ./internal/modules/generic/commands_test.go

Command: type
Module: constants.GenericModule
Categories: constants.KeyspaceCategory, constants.ReadCategory, constants.FastCategory
Description: (TYPE key) Returns the string representation of the type of the value stored at key. The different types that can be returned are: string, integer, float, list, set, zset, hash and stream.
Sync: false

Embedded Spec:

Command File: ./echovault/api_generic.go
Test File: ./echovault/api_generic_test.go

Implement the SSCAN command

Scan the elements of a set.
Reference: https://redis.io/docs/latest/commands/sscan/

Client-Server Spec:

Command File: ./internal/modules/set/commands.go
Test File: ./internal/modules/set/commands_test.go

Command: sscan
Module: constants.SetModule
Categories: contants.SetCategory, constants.ReadCategory, constants.SlowCategory
Description: (SSCAN key cursor [MATCH pattern] [COUNT count]) scan the elements of a set
Sync: false

Embedded Spec:

Command File: ./echovault/api_set.go
Test File: ./echovault/api_set_test.go

Documentation

Add documentation to ./docs/docs/commands/set/sscan.mdx

Implement sharding support using consistent hashing

Allow the user to set up a shards using consistent hashing.

This feature should support:

1. Shards containing multiple single nodes.
2. Shards containing multiple replication clusters.

Reference: https://github.com/buraksezer/consistent

Implement DECRBY command

Implement the DECRBY command, which decrements the value at the specified key by the specified amount if the value is an integer or a float.
Reference: https://redis.io/docs/latest/commands/decrby/

Client-Server Spec:

Command File: ./internal/modules/generic/commands.go
Test File: ./internal/modules/generic/commands_test.go

Command: decrby
Module: constants.GenericModule
Categories: constants.WriteCategory, constants.FastCategory
Description: (DECRBY key decrement) The DECRBY command reduces the value stored at the specified key by the specified decrement. If the key does not exist, it is initialized with a value of 0 before performing the operation. If the key's value is not of the correct type or cannot be represented as an integer, an error is returned.
Sync: true

Embedded Spec:

Command File: ./echovault/api_generic.go
Test File: ./echovault/api_generic_test.go

Implement GETDEL command

Implement the GETDEL command, which gets the current value at the specified key and then delete the key.
Reference: https://redis.io/docs/latest/commands/getdel/

Client-Server Spec:

Command File: ./internal/modules/generic/commands.go
Test File: ./internal/modules/generic/commands_test.go

Command: getdel
Module: constants.GenericModule
Categories: constants.WriteCategory, constants.FastCategory
Description: (GETDEL key) Get the value of key and delete the key. This command is similar to [GET](https://redis.io/docs/latest/commands/get/), except for the fact that it also deletes the key on success (if and only if the key's value type is a string).
Sync: true

Embedded Spec:

Command File: ./echovault/api_generic.go
Test File: ./echovault/api_generic_test.go

go get produces incorrect go module path

Seems as your go module name is lowercased so in your ReadMe downloading via go get... returns this error.

go: github.com/echoVault/echoVault@latest (v0.8.0) requires github.com/echoVault/[email protected]: parsing go.mod:
        module declares its path as: github.com/echovault/echovault
                but was required as: github.com/echoVault/echoVault

Use RESP AOF & Preamble for RAFT layer logging and snapshotting instead of depending on boltdb

Use aof package in the raft module instead of BoltDB for logging and snapshotting

Create test suite for AOF Engine

Create a test suite for the Custom AOF Engine.

Implement RENAMENX command

Implement the RENAMENX command, which renames the specified key with the new name only if the new keyname does not already exist.
Reference: https://redis.io/docs/latest/commands/renamenx/

Client-Server Spec:

Command File: ./internal/modules/generic/commands.go
Test File: ./internal/modules/generic/commands_test.go

Command: renamenx
Module: constants.GenericModule
Categories: constants.KeyspaceCategory, constants.WriteCategory, constants.FastCategory
Description: (RENAMENX key newkey) Renames the specified key with the new name only if the new name does not already exist.
Sync: true

Embedded Spec:

Command File: ./echovault/api_generic.go
Test File: ./echovault/api_generic_test.go

NOTE: You may have to edit the HandlerFuncParams type to include a function that renames keys. Alternatively, you could use the GetValues, DeleteKeys and SetValues methods together to delete the old key and set a new key with the old value. The first approach would be more atomic though.

At the moment, only replication clusters exist without sharding. So the key should be renames across the entire cluster.

Implement RANDOMKEY command

Implement the RANDOMKEY command, which returns a random key from echovault (no database selection set yet).
Reference: https://redis.io/docs/latest/commands/randomkey/

Client-Server Spec:

Command File: ./internal/modules/generic/commands.go
Test File: ./internal/modules/generic/commands_test.go

Command: randomkey
Module: constants.GenericModule
Categories: constants.KeyspaceCategory, constants.ReadCategory, constants.SlowCategory
Description: (RANDOMKEY) Returns a random key.
Sync: false

Embedded Spec:

Command File: ./echovault/api_generic.go
Test File: ./echovault/api_generic_test.go

NOTE: You may have to edit the HandlerFuncParams type to include a function that returns a random key.

Implement INCRBY command

Implement the INCRBY command, which increments the value at the specified key by the specified amount if the value is an integer or a float.
Reference: https://redis.io/docs/latest/commands/incrby/

Client-Server Spec:

Command File: ./internal/modules/generic/commands.go
Test File: ./internal/modules/generic/commands_test.go

Command: incrby
Module: constants.GenericModule
Categories: constants.WriteCategory, constants.FastCategory
Description: (INCRBY key increment) Increments the number stored at key by increment. If the key does not exist, it is set to 0 before performing the operation. An error is returned if the key contains a value of the wrong type or contains a string that can not be represented as integer.
Sync: true

Embedded Spec:

Command File: ./echovault/api_generic.go
Test File: ./echovault/api_generic_test.go

Differentiate Read and Write Keys when Authorising commands in ACL Layer

The ACL layer uses a command's KeyExtractionFunc to extract all the keys the command accesses.

KeyExtractionFunc returns a slice containing all the keys (read and write). However, the ACL layer cannot differentiate which keys are read and which ones are write.

This forces the ACL layer to check all returned keys against all allowed/excluded read and write keys.

TODO:

KeyExtractionFunc should return a struct containing a slice of read keys and a slice of write keys to allow the ACL layer to conduct a more granular check.

Create a production level dockerfile

Create a dockerfile for prod deployment. This dockerfile will be used to build the container that people will pull from dockerhub.

Implement SCAN command

Scans the keys in the currently selected database.
Reference: https://redis.io/docs/latest/commands/scan/

Client-Server Spec:

Command File: ./internal/modules/generic/commands.go
Test File: ./internal/modules/generic/commands_test.go

Command: scan
Module: constants.GenericModule
Categories: contants.KeyspaceCategory, constants.ReadCategory, constants.SlowCategory
Description: (SCAN cursor [MATCH pattern] [COUNT count] [TYPE type]) scan the keys in the currently selected database
Sync: false

Embedded Spec:

Command File: ./echovault/api_generic.go
Test File: ./echovault/api_generic_test.go

Documentation

Add documentation to ./docs/docs/commands/generic/scan.mdx

Create Godoc documentation for Embedded API

Created Godoc documentation for publishing a GO-embedded library on the v0.3.0 release.

Implement INCR command

Implement the INCR command, which increments the value at the specified key by 1 if the value is an integer or a float.
Reference: https://redis.io/docs/latest/commands/incrby/

Client-Server Spec:

Command File: ./internal/modules/generic/commands.go
Test File: ./internal/modules/generic/commands_test.go

Command: incr
Module: constants.GenericModule
Categories: constants.WriteCategory, constants.FastCategory
Description: (INCR key) Increments the number stored at key by one. If the key does not exist, it is set to 0 before performing the operation. An error is returned if the key contains a value of the wrong type or contains a string that can not be represented as an integer.
Sync: true

Embedded Spec:

Command File: ./echovault/api_generic.go
Test File: ./echovault/api_generic_test.go

Snapshot On Separate Process

At the moment, EchoVault copies the entire store and then runs a snapshot on a goroutine in the same process. This has the effect of (almost) doubling the memory usage of the current process when the entire store is copied.

The idea is to create a new EchoVault process using exec.Command, passing the state to this new process and creating the snapshot in the new process. This should prevent the current process from blowing up in memory usage.

Implement TOUCH command

Implement the TOUCH command, which updates the access times of the provided keys.
Reference: https://redis.io/docs/latest/commands/touch/

Client-Server Spec:

Command File: ./internal/modules/generic/commands.go
Test File: ./internal/modules/generic/commands_test.go

Command: touch
Module: constants.GenericModule
Categories: constants.KeyspaceCategory, constants.ReadCategory, constants.FastCategory
Description: (TOUCH key [key ...]) Alters the last access time of a key(s). A key is ignored if it does not exist.
Sync: true

Embedded Spec:

Command File: ./echovault/api_generic.go
Test File: ./echovault/api_generic_test.go

NOTE: You may have to edit the HandlerFuncParams type to include a function that touches the keys.

Implement ECHO command

Implement the ECHO command, which returns the message to the connection.
Reference: https://redis.io/docs/latest/commands/echo/

Client-Server Spec:

Command File: ./internal/modules/connection/commands.go
Test File: ./internal/modules/connection/commands_test.go

Command: echo
Module: constants.ConnectionModule
Categories: constants.ConnectionCategory, constants.FastCategory
Description: (ECHO message) Returns message.
Sync: false

Skip ACL Authorisation for Embedded API Calls

The handleCommand method checks if the caller is authorized to execute the provided command. This makes sense for clients connected over TCP. However, this check must be skipped for embedded clients as there's no need to authorize embedded API call commands.

Implement APPEND command

Implement the APPEND command, which appends the provided string to the value at the specified key if the value is a string.
Reference: https://redis.io/docs/latest/commands/append/

Client-Server Spec:

Command File: ./internal/modules/string/commands.go
Test File: ./internal/modules/string/commands_test.go

Command: append
Module: constants.StringModule
Categories: constants.StringCategory, constants.WriteCategory, constants.FastCategory
Description: (APPEND key value) If key already exists and is a string, this command appends the value at the end of the string. If key does not exist it is created and set as an empty string, so APPEND will be similar to [SET] in this special case.
Sync: true

Embedded Spec:

Command File: ./echovault/api_string.go
Test File: ./echovault/api_string_test.go

build does not work

How can I run this project and make it reproducible?

error output

cd /root; cd octavia/diskimage-create/; ./diskimage-create.sh -a amd64 -d jammy -i ubuntu-minimal -o amphora-x64-haproxy-jammy.qcow2 -s 3

# go version
go version go1.22.2 linux/amd64

# git log -p -2
commit 2528082d4158680c7b59d7e526aec2b62885d8c4 (HEAD -> main, origin/main, origin/HEAD)
Author: Kelvin Mwinuka <[email protected]>
Date:   Thu Apr 11 11:01:45 2024 +0800

    Update README.md

diff --git a/README.md b/README.md
index 33bd7bd..e581f7a 100644
--- a/README.md
+++ b/README.md
@@ -7,9 +7,6 @@
 <br/>
 [![Go Reference](https://pkg.go.dev/badge/github.com/echovault/echovault.svg)](https://pkg.go.dev/github.com/echovault/echovault)
 <br/>
-[![Discord](https://img.shields.io/discord/1211815152291414037?style=flat&label=Discord&color=%235865F2)
-](https://discord.gg/aasBFwDm)
-<br/>
 <hr/>

 <img alt="echovault_logo" src="./images/EchoVault GitHub Cover.png" width="5062" />

commit c6b246efa455caaea53a6ba7a83115db826d36e1
Merge: 9b7d93f 64bbda6
Author: Kelvin Mwinuka <[email protected]>
Date:   Sat Apr 6 02:51:27 2024 +0800

    Merge pull request #21 from EchoVault/chore/fix-race-condition

    Fixed race conditions in acl module tests
#

# make build
env CC=x86_64-linux-musl-gcc GOOS=linux GOARCH=amd64 DEST=bin/linux/x86_64 make build-server
make[1]: вход в каталог «/root/EchoVault»
CC=x86_64-linux-musl-gcc GOOS=linux GOARCH=amd64 go build -o bin/linux/x86_64/server ./cmd/main.go
# runtime/cgo
cgo: C compiler "x86_64-linux-musl-gcc" not found: exec: "x86_64-linux-musl-gcc": executable file not found in $PATH
make[1]: *** [Makefile:2: build-server] Ошибка 1
make[1]: выход из каталога «/root/EchoVault»
make: *** [Makefile:5: build] Ошибка 2

Implement KEYS command

Implement the KEYS command, which returns all the keys that match the provided pattern.
Reference: https://redis.io/docs/latest/commands/keys/

Client-Server Spec:

Command File: ./internal/modules/generic/commands.go
Test File: ./internal/modules/generic/commands_test.go

Command: keys
Module: constants.GenericModule
Categories: constants.KeyspaceCategory, constants.ReadCategory, constants.SlowCategory, constants.DangerousCategory
Description: (KEYS pattern) Returns an array of keys that match the provided glob pattern.
Sync: false

Embedded Spec:

Command File: ./echovault/api_generic.go
Test File: ./echovault/api_generic_test.go

NOTE: You may have to edit the HandlerFuncParams type to include a function that returns all matching keys.

Update Dockerfile.dev to use multi stage build

The current state of Dockerfile.dev requires developers to first build the binaries locally, which are then copied into the image during the docker build.

Instead we should update the Dockerfile.dev to use a multi stage build process to allow for the full build process to take place within docker.

Create test suite for snapshot engine

Create test suite for custom snapshot engine.

docker feedback

(saw your post on discord, grats/nice)

Some feedback:

in Dockerfile.prod you have FROM homebrew/brew:latest

Have you swapped accidentally .prod and .dev?

And for whichever ends up the real prod:

Take a Quick Look at https://laurentsv.com/blog/2024/06/25/stop-the-go-and-docker-madness.html

ie you can reduce your docker final image a lot (FROM scratch ftw)

ps: while I'm at it... why cmd/main.go ... move it up?

Allow configuration of consistency policy

EchoVault is currently strongly consistent. This change will aim to make this policy configurable.

At the moment, echovault has a memberlist layer and a raft layer. The memberlist layer is mainly used to establish a cluster skeleton, essentially the "chassis" of the cluster. Memberlist nodes will communicate with other nodes to ask permission to join the raft cluster using the gossip protocol. Once the gossiped message reaches the raft cluster leader, the leader will add the node to the raft cluster.

We must allow a user to configure this functionality using the config flags. The option will enable the user to forgo the raft layer and use the memberlist layer for gossip-based state propagation to allow for eventual consistency. Otherwise, the user can opt for the strong consistency of the raft layer.

RAFT

I need to run within single node and multi-node environment without reconfiguring. The following messages occur during single node environment and no caching is happening

2024-06-26T16:32:52.327-0400 [INFO] raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:01J1B3A78G06A0HWM86QZYAVB8 Address:localhost:7481}]"

2024-06-26T16:32:52.327-0400 [INFO] raft: entering follower state: follower="Node at 127.0.0.1:7481 [Follower]" leader-address= leader-id=

2024-06-26T16:32:55.204-0400 [WARN] raft: heartbeat timeout reached, not part of a stable configuration or a non-voter, not triggering a leader election

2024/06/26 17:11:14 deleted key 3491595662
2024/06/26 17:11:14 1 keys sampled, 1 keys deleted
2024/06/26 17:11:14 deletion ratio (100 percent) reached threshold (20 percent), sampling again

can you explain EvictionInterval, EvictionSample, and SET EX and how they relate to each other.
i have 64mb cache allocated, and the cache contain 2 messages of 1024. Please explain deletion ratio (100 percent) reached threshold (20 percent), sampling again

SetOptions and ExpireOptions should be an interface, not struct

Disclaimer: This is just a suggestion, I think it's a good idea but I if no one else does I'm not gonna fight super hard for it

Problem:
Currently, in the generic API we define structs to allow users to set expiry and other behaviors for Set, Expire, ExpireAt, PExpire, and PExpireAt methods. Boolean fields are set by the user, but there are no guardrails in place to stop a user from passing in something like ExpireOptions{NX: true, LT: true}.

I don't think it's a major issue, but I do think that this design could result in users experiencing what they perceive as unexpected behaviors.

Proposed Solution:
I know go enums are nonexistent and custom implementation is a bit of a pain point for the language, however, I believe something like this would still be readable and provide a better experience to users.

type ExOpt string

const (
	NX ExOpt = "NX"
	XX ExOpt = "XX"
	LT ExOpt = "LT"
	GT ExOpt = "GT"
)

type ExpireOptions interface {
	IsExOpt() ExOpt
}

func (x ExOpt) IsExOpt() ExOpt { return x }


// to show function arg
func printConsts(str ExpireOptions) {

	fmt.Println(str)
}

This allows stronger type safety for methods that currently use these structs, and more clear expected behavior for users implementing these methods. Additionally, this provides use of LSPs (tested with gopls) to discover available constants in the package by navigating to refrence/definition. This coupled with inline documentation should provide a better user experience in my opinion.

https://go.dev/play/p/a3PeonPSbkk

This would require some refactoring on the methods that use these structs, but it would simplify the logic from using switch cases to simply passing the constant into the command.

Extend EchoVault commands with embedded API

Expose a method in the embedded api to add a new command. This is similar to extension via .so files.

Create test for pubsub module

Test all the command handlers in the pubsub module.

Server panics when connecting with `redis-cli`

Hello, I am trying out this project because it looks cool and I'd like to learn more about Raft. If you're open to it, I might be interested in implementing some of the Redis commands that aren't yet supported, as a way to learn :)

When running the hello world server (from the README) and connecting with redis-cli, the server panics:

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0xbd1982]

goroutine 10 [running]:
github.com/echovault/echovault/echovault.(*EchoVault).handleCommand(0xc000181b88, {0xde2418, 0xc0001fc5d0}, {0xc0002c6000, 0x11, 0x2000}, 0xc000025610, 0x0, 0x0)
        /home/b/go/pkg/mod/github.com/echovault/[email protected]/echovault/modules.go:110 +0x702
github.com/echovault/echovault/echovault.(*EchoVault).handleConnection(0xc000181b88, {0xde5dd0, 0xc0000a4500})
        /home/b/go/pkg/mod/github.com/echovault/[email protected]/echovault/echovault.go:445 +0x2c8
created by github.com/echovault/echovault/echovault.(*EchoVault).startTCP in goroutine 1
        /home/b/go/pkg/mod/github.com/echovault/[email protected]/echovault/echovault.go:415 +0x605

I think this is a bug because redis-cli should be using RESP, and the README says that echovault is RESP-compatible?

Server code:

func main() {
	server, err := echovault.NewEchoVault() // echovault.WithConfig(cfg))
	if err != nil {
		log.Fatal(err)
	}
	server.Start()
}

The same error happens if I run the included CLI with echovault -data-dir /tmp/echovault -bind-addr 127.0.0.1