microsoft / garnet Goto Github PK

this member is called here

which this API is called from the public API TryWriteKeyValueSpanByte here

which takes ref SpanByte.

It is a GC hole since it's never pinned or otherwise fixed.

This is also a GC hole, since the memory is not fixed (you can also just use Unsafe.SizeOf here):

This is also seems to be a GC hole, since the memory is never fixed on this public API:

(note this list is not exhaustive)

Here's your opportunity to implement a RESP command in Garnet!

Syntax:
ZMSCORE key member [member ...]

Command description:
Returns the scores associated with the specified members in the sorted set stored at key.

For every member that does not exist in the sorted set, a nil value is returned.

Response:
One of the following:

Nil reply: if the member does not exist in the sorted set.
Array reply: a list of string member scores as double-precision floating point numbers.

How to approach this task:

1. Add the new command info to the sortedSetCommandsInfoMap in the RespCommandsInfo class (Garnet.server/Resp/RespCommandsInfo.cs)
2. Add the new command string to the returned HashSet in the RespInfo.GetCommands method (Garnet.server/Resp/RespInfo.cs)
3. Add the appropriate fast parsing logic for the new command in the RespCommand.FastParseArrayCommand method (use other commands as reference) (Garnet.server/Resp/RespCommand.cs)
4. Implement the wrapper method to the storage layer in StorageSession (Garnet.server/Storage/Session/ObjectStore/SortedSetOps.cs)
5. Define a new method in IGarnetAPI and implement it in GarnetApiObjectCommands, calling the method you have implemented in step #4 (Garnet.server/API/IGarnetAPI.cs, Garnet.server/API/GarnetApiObjectCommands.cs)
6. Add a new method to the RespServerSession class which will call the storage API and return the appropriate response (Garnet.server/Resp/Objects/SortedSetCommands.cs)
7. Add a new method to SortedSetObjImpl which will perform the required operation on the hash object (Garnet.server/Objects/SortedSet/SortedSetObjectImpl.cs)
8. Add a new enum value to SortedSetOperation and add the appropriate case to the switch in SortedSetObject.Operate, in which you will call the method defined in step #8 (Garnet.server/Objects/SortedSet/SortedSetObject.cs)
9. Add an appropriate case to the switch in TransactionManager.SortedSetObjectKeys (Garnet.server/Transaction/TxnKeyManager.cs)
10. Add tests for the new command in the RespSortedSetTests class (Garnet.test/RespSortedSetTests.cs)

Tip: First add a simple test that calls the new command and use it to debug as you develop, it will make your life much easier!

If you have any questions, the Garnet team is here to help!

Hi!
I can run a console app locally (Windows) fine, Just the three lines in Program.cs:

            using var server = new Garnet.GarnetServer(["--port", "8080"]);
            server.Start();
            Thread.Sleep(Timeout.Infinite);

But when packaging this console app and running it in kubernetes (using alpine, net8 as base) I get

Unable to initialize server due to exception: Exception of type 'System.OutOfMemoryException' was thrown.

The pod has 2 GB memory, which should be more than sufficient. And I can see from monitoring that the pod hardly uses any memory at all.

Any guidance on how to proceed?

NOTE: Currently we do not support leader election on failover because we assume replicas cannot initiate failover on their own

Issuing CLUSTER FAILOVER without any arguments should make the old primary a replica to the new primary.
This is currently not happening.
Our initial assumption was that CLUSTER FAILOVER will always be called with TAKEOVER option since the primary will not be reachable.
We did not account for planned failovers that are still a valid case.
In addition, the implementation of attachReplicas is buggy and may throw an exception if any one of the remote nodes is unreachable, resulting in the reachable nodes never being informed of the primary change.

Proposed solution:

• Refactor attachReplicas so catch statement is inside the loop and does not interfere with communication to remote nodes in the event of an exception.
• Make old primary replica of the new primary when CLUSTER FAILOVER is issued with default option

NOTE: Avoid changing information directly owned remote nodes by changing the local configuration
In the failover case, if we make remote nodes replicas to the new primary by directly updating the local configuration, we will put the cluster into an inconsistent state if those nodes are unreachable.
We really, on issuing requests to remote nodes to change their own state, so we can make sure that the change is acknowledged if when the two nodes communicating are well connected.
The exception to this rule is when changing ownership of slots as in the failover case.
We cannot avoid doing this because slots can be owned by any instance at any given point in time.
For the latter scenario, we should be extremely careful to avoid any split-brain inconsistencies.
At least until leader election is properly implemented.

Hi,
The PX command does not seem to be supported.

Garnet reports ERR unknown command when I am using SET k1 v1 PX 10000.

Congrats on the release of this awesome project.

Given the project came out of Microsoft Research team, wondering is it ready to use in production?

Primarily, I wanted to know whether it is already being used in some form in production within Microsoft?

Noticed it was enabled:

There's the docosaurus docs, and the wiki. Is there an objective for the wiki, or just leftover?

Uncertain if you want to prioritize this bug, but I think it would help optics around compatibility w.r.t Redis 7.

Reproduction steps:

# clone on macOS
docker-compose up -d

# expected:
brew install redis
brew services start redis
redis-benchmark -p 6379

# unexpected result:
redis-benchmark -p 3278
...
Error: Server closed the connection

In docker-compose logs -f observe:

ProcessMessages threw exception System.ArgumentException: Range of random number does not contain at least one possibility.
	at System.Security.Cryptography.RandomNumberGenerator.GetInt32(Int32 fromInclusive, Int32 toExclusive)
	at Garnet.server.SetObject.SetPop(Byte* input, Int32 length, SpanByteAndMemory& output) in /source/libs/server/Objects/Set/SetObjectImpl.cs:line 191
	at Garnet.server.SetObject.Operate(SpanByte& input, SpanByteAndMemory& output, Int64& sizeChange) in /source/libs/server/Objects/Set/SetObject.cs:line 121
	at Garnet.server.StorageSession.RMWObjectStoreOperationWithOutput[TObjectContext](Byte[] key, ArgSlice input, TObjectContext& objectStoreContext, GarnetObjectStoreOutput& outputFooter) in /source/libs/server/Storage/Session/ObjectStore/Common.cs:line 52
	at Garnet.server.StorageSession.SetPop[TObjectContext](Byte[] key, ArgSlice input, GarnetObjectStoreOutput& outputFooter, TObjectContext& objectContext) in /source/libs/server/Storage/Session/ObjectStore/SetOps.cs:line 431
	at Garnet.server.GarnetApi`2.SetPop(Byte[] key, ArgSlice input, GarnetObjectStoreOutput& outputFooter) in /source/libs/server/API/GarnetApiObjectCommands.cs:line 289
	at Garnet.server.RespServerSession.SetPop[TGarnetApi](Int32 count, Byte* ptr, TGarnetApi& storageApi) in /source/libs/server/Resp/Objects/SetCommands.cs:line 392
	at Garnet.server.RespServerSession.ProcessArrayCommands[TGarnetApi](TGarnetApi& storageApi) in /source/libs/server/Resp/RespServerSession.cs:line 376
	at Garnet.server.RespServerSession.ProcessBasicCommands[TGarnetApi](Byte* ptr, RespCommand cmd, TGarnetApi& storageApi) in /source/libs/server/Resp/RespServerSession.cs:line 327
	at Garnet.server.RespServerSession.ProcessMessages() in /source/libs/server/Resp/RespServerSession.cs:line 273
	at Garnet.server.RespServerSession.TryConsumeMessages(Byte* reqBuffer, Int32 bytesReceived) in /source/libs/server/Resp/RespServerSession.cs:line 199

It appears some of the code doesn't account for the range [0, 0) which GetInt32 rejects, see:

• garnet/libs/server/Objects/Set/SetObjectImpl.cs

  Line 191 in 5b6c3ae

  int index = RandomNumberGenerator.GetInt32(0, set.Count);

• https://learn.microsoft.com/en-us/dotnet/api/system.security.cryptography.randomnumbergenerator.getint32?view=net-8.0

Want me to attempt a PR that patches this? Looks like a bug in the SPOP implementation. https://redis.io/commands/spop/

Hi, I can't find information about available eviction policies used when maxmemory is reached. Is LRU supported?

Your docker image doesn't work on Ubuntu hosts. I get /app/GarnetServer: no such file or directory despite this file existing in the container. Out of curiosity I exec-ed inside the container and I see server executable in that location. Launching it manually in the terminal gives /app/GarnetServer: not found. So something is not right.

My project uses Microsoft.Extensions.Caching.StackExchangeRedis as the implementation of distributed cache. When connected to Garnet, the Set action will return an ERR unknown command exception.

IDistributedCache distributedCache

distributedCache.SetString(key, value, new DistributedCacheEntryOptions { AbsoluteExpirationRelativeToNow = absoluteExpirationRelativeToNow });

ERR unknown command

http://Iheyreally.comhttp://www.IHeyReally.org

using the tools named "security-code-scan" to scan the solution ccode has found the security error SCS0005(Weak random number generator). this cs file and position list :

1. libs\common\Generator.cs (24,13)
2. libs\server\Objects\Set\SetObjectImpl.cs (181,37)、(193,33):
3. libs\server\Objects\Hash\HashObjectImpl.cs (273,46)、(275,60)、(299,33)
4. libs\server\Objects\SortedSet\SortedSetObjectImpl.cs (683,38)、(686,52)
5. other test/benchmark/playground files.

Here's your opportunity to implement a couple of RESP commands in Garnet!

SUNION

Syntax:
SUNION key [key ...]

Command description:
Returns the members of the set resulting from the union of all the given sets.

For example:

key1 = {a,b,c,d}
key2 = {c}
key3 = {a,c,e}
SUNION key1 key2 key3 = {a,b,c,d,e}

Keys that do not exist are considered to be empty sets.

Response:

Array reply: a list with members of the resulting set.

How to approach this task:

1. Add the new command info to the setCommandsInfoMap in the RespCommandsInfo class (Garnet.server/Resp/RespCommandsInfo.cs)
2. Add the new command string to the returned HashSet in the RespInfo.GetCommands method (Garnet.server/Resp/RespInfo.cs)
3. Add the appropriate fast parsing logic for the new command in the RespCommand.FastParseArrayCommand method (use other commands as reference) (Garnet.server/Resp/RespCommand.cs)
4. Implement the wrapper method to the storage layer in StorageSession (Garnet.server/Storage/Session/ObjectStore/SetOps.cs)
5. Define a new method in IGarnetAPI and implement it in GarnetApiObjectCommands, calling the method you have implemented in step #4 (Garnet.server/API/IGarnetAPI.cs, Garnet.server/API/GarnetApiObjectCommands.cs)
6. Add a new method to the RespServerSession class which will call the storage API and return the appropriate response (Garnet.server/Resp/Objects/SetCommands.cs)
7. Add a new method to SetObjImpl which will perform the required operation on the hash object (Garnet.server/Objects/Set/SetObjectImpl.cs)
8. Add a new enum value to SetOperation and add the appropriate case to the switch in SetObject.Operate, in which you will call the method defined in step #8 (Garnet.server/Objects/Set/SetObject.cs)
9. Add an appropriate case to the switch in TransactionManager.SetObjectKeys (Garnet.server/Transaction/TxnKeyManager.cs)
10. Add tests for the new command in the RespSetTests class (Garnet.test/RespSetTests.cs)

Tip: First add a simple test that calls the new command and use it to debug as you develop, it will make your life much easier!

If you have any questions, the Garnet team is here to help!

Hi all, are you planning on creating an IDistributedCache implementation anytime soon?

Since Garnet has been announced as being protocol-compatible with Redis I thought it would've worked right away by using the standard Redis implementation (see here), but after actually playing with it I discovered that Garnet does not currently support LUA scripting:

3. Garnet does not support Lua scripting. We have an experimental version, but it was noted to be too slow for realistic use so we have not added it to the project.

LUA scripting is what has been used in the aforementioned impl (see here) and therefore without it it's not possible to use Garnet this way.

Having a working impl of IDistributedCache for Garnet would drive the adoption a lot, since that is the standard way of using a distributed cache in .NET.

Also, that would unlock other great possibilities like being able to use Garnet in projects that use IDistributedCache as a "lingua franca", like FusionCache (mine) which is compatible with any impl of such interface (here for more).

All in all I think that, if the effort for such impl is not huge, that is something that you should consider to skyrocket the usage.

Thoughts?

ps: maybe @mgravell can give us some hints about such approach? Are there any hidden pitfalls that should be looked at?

Has any testing been done to ensure fully compatible with Twemproxy?

This is exciting, thanks!

Hi! I wrote some dirty benchmark code and observed that with an increased number of concurrent in-flight queries using the StringGet method with the callback, the chances for it to hang increases. During the hang I observe high CPU usage on the machine where the test code is executed and the execution is somewhere within StringGet. That's all I know now :-)

There are chances that StringSet might hang or complete very slowly, too.

I think some internal structure in Garnet client is getting overloaded, or there's maybe an exponential complexity somewhere in the client kicking in, not sure about that. This way or another, increasing the value of NUM_TASKS in the code below makes the hang more likely, while decreasing it definitely improves the probability of timely and performant execution.

Please check it and fix if necessary.

This code is copied from Visual Studio and is using top-level statements.

/*
Launch Garnet server with:
$ cd main\GarnetServer
$ dotnet run -c Release -f net8.0
*/

using Garnet.client;
using System.Diagnostics;

Console.WriteLine("Connecting to Garnet");

using GarnetClient client = new GarnetClient("127.0.0.1", 3278);

client.Connect();

await client.ExecuteForStringResultAsync("FLUSHDB");

Console.WriteLine("Inserting data");

//int NUM_TASKS = Environment.ProcessorCount / 2;
int NUM_TASKS = 24;
var tasks = new List<Task>();

const int NUM_KEYS = 1_000_000;
const string VALUE_TO_INSERT = "value1";
long globalCounter = 0;

Stopwatch sw = Stopwatch.StartNew();

for (int task_id = 0; task_id < NUM_TASKS; task_id++)
{
    var task = Task.Factory.StartNew(task_id_obj =>
    {
        CountdownEvent cdeForSetOperation = new CountdownEvent(NUM_KEYS);
        int local_task_id = (int)task_id_obj!;

        for (int i = 0; i < NUM_KEYS; i++)
        {
            string key = $"key_{local_task_id}_{i}";
            client.StringSet(key, VALUE_TO_INSERT, null);
            cdeForSetOperation.Signal();
        }

        cdeForSetOperation.Wait();
        Interlocked.Add(ref globalCounter, NUM_KEYS);
    }, task_id);
    tasks.Add(task);
}

Task.WaitAll(tasks.ToArray());

sw.Stop();
Console.WriteLine($"Insertion completed in {sw.Elapsed}, iterations per seconds: {decimal.Round(1000m * NUM_KEYS * NUM_TASKS / sw.ElapsedMilliseconds, 1)}, inserted {globalCounter} values");

Console.WriteLine("Querying data");

tasks.Clear();
await Task.Delay(1000);

sw = Stopwatch.StartNew();
const int NUM_ITERATIONS = 1_000_000;

// A dummy counter to do something with the value from the db
long totalLength = 0;

for (int task_id = 0; task_id < NUM_TASKS; task_id++)
{
    var task = Task.Factory.StartNew(() =>
    {
        long length = 0;
        CountdownEvent cdeForQuery = new CountdownEvent(NUM_ITERATIONS);

        for (int i = 0; i < NUM_ITERATIONS; i++)
        {
            client.StringGet("key_0_0", (c, v) =>
            {
                Interlocked.Add(ref length, v?.Length ?? 0);
                cdeForQuery.Signal();
            });
        };

        cdeForQuery.Wait();
        Interlocked.Add(ref totalLength, length);
    });

    tasks.Add(task);
}

Task.WaitAll(tasks.ToArray());

sw.Stop();
Console.WriteLine($"Querying completed in {sw.Elapsed}, iterations per seconds: {decimal.Round(1000m * NUM_ITERATIONS * NUM_TASKS / sw.ElapsedMilliseconds, 1)}, total length = {totalLength}, {(totalLength == NUM_TASKS * NUM_ITERATIONS * VALUE_TO_INSERT.Length ? "ok" : "NOT OK")}");

await client.ExecuteForStringResultAsync("FLUSHDB");

[email protected]/http://iheyreally.org > _ No description provided. _

#83

Originally posted by @Heyitsquoracom in #83 (comment)

[email protected]

Good evening, would you like to ask if Garnet has any plans for support vector storage in the future? Like Redis, it feels amazing!

In memurai-cli, when running the SCAN command with no parameters, the session just hangs in Garnet. In Redis, this message comes back: (error) ERR wrong number of arguments for 'scan' command. This should be the expected behavior.

Using Garnet version 1.0.1, compiled in VS2022 in Release/Any CPU mode and using the .NET 8 version.

Garnet:

Redis:

Does Garnet support persistence?

Running docker compose up modified for ubuntu-x64 or not modified (result is the same) build fails.

Steps to reproduce:

• checkout git repo
• run docker compose up

The build error output:

Step 1/12 : FROM mcr.microsoft.com/dotnet/sdk:8.0 AS build
...
Step 5/12 : RUN dotnet build -c Release
---> Running in 818581752e2a
MSBuild version 17.9.6+a4ecab324 for .NET
Determining projects to restore...
All projects are up-to-date for restore.
Garnet.common -> /source/libs/common/bin/AnyCPU/Release/net7.0/Garnet.common.dll
Garnet.common -> /source/libs/common/bin/AnyCPU/Release/net6.0/Garnet.common.dll
Garnet.common -> /source/libs/common/bin/AnyCPU/Release/net8.0/Garnet.common.dll

...
Resp.benchmark -> /source/benchmark/Resp.benchmark/bin/Release/net6.0/Resp.benchmark.dll
/source/playground/ClusterStress/OnlineReqGen.cs(6,1): error IDE0005: Using directive is unnecessary. (https://learn.microsoft.com/dotnet/fundamentals/code-analysis/style-rules/ide0005) [/source/playground/ClusterStress/ClusterStress.csproj::TargetFramework=net6.0]
Garnet.host -> /source/libs/host/bin/AnyCPU/Release/net7.0/Garnet.host.dll
Embedded.perftest -> /source/playground/Embedded.perftest/bin/Release/net7.0/Embedded.perftest.dll
GarnetServer -> /source/main/GarnetServer/bin/AnyCPU/Release/net7.0/GarnetServer.dll
Garnet.server -> /source/libs/server/bin/AnyCPU/Release/net6.0/Garnet.server.dll
Garnet.test.cluster -> /source/test/Garnet.test.cluster/bin/AnyCPU/Release/net7.0/Garnet.test.cluster.dll
Garnet.test -> /source/test/Garnet.test/bin/AnyCPU/Release/net7.0/Garnet.test.dll
Garnet.host -> /source/libs/host/bin/AnyCPU/Release/net8.0/Garnet.host.dll
GarnetServer -> /source/main/GarnetServer/bin/AnyCPU/Release/net8.0/GarnetServer.dll
Garnet.test.cluster -> /source/test/Garnet.test.cluster/bin/AnyCPU/Release/net8.0/Garnet.test.cluster.dll
Garnet.cluster -> /source/libs/cluster/bin/AnyCPU/Release/net6.0/Garnet.cluster.dll
Embedded.perftest -> /source/playground/Embedded.perftest/bin/Release/net8.0/Embedded.perftest.dll
Garnet.host -> /source/libs/host/bin/AnyCPU/Release/net6.0/Garnet.host.dll
Garnet.test -> /source/test/Garnet.test/bin/AnyCPU/Release/net8.0/Garnet.test.dll
Garnet.test -> /source/test/Garnet.test/bin/AnyCPU/Release/net6.0/Garnet.test.dll
TstRunner -> /source/playground/TstRunner/bin/Release/net7.0/TstRunner.dll
GarnetServer -> /source/main/GarnetServer/bin/AnyCPU/Release/net6.0/GarnetServer.dll
Embedded.perftest -> /source/playground/Embedded.perftest/bin/Release/net6.0/Embedded.perftest.dll
Garnet.test.cluster -> /source/test/Garnet.test.cluster/bin/AnyCPU/Release/net6.0/Garnet.test.cluster.dll

Build FAILED.

/source/playground/ClusterStress/OnlineReqGen.cs(6,1): error IDE0005: Using directive is unnecessary. (https://learn.microsoft.com/dotnet/fundamentals/code-analysis/style-rules/ide0005) [/source/playground/ClusterStress/ClusterStress.csproj::TargetFramework=net7.0]
/source/playground/ClusterStress/OnlineReqGen.cs(6,1): error IDE0005: Using directive is unnecessary. (https://learn.microsoft.com/dotnet/fundamentals/code-analysis/style-rules/ide0005) [/source/playground/ClusterStress/ClusterStress.csproj::TargetFramework=net8.0]
/source/playground/ClusterStress/OnlineReqGen.cs(6,1): error IDE0005: Using directive is unnecessary. (https://learn.microsoft.com/dotnet/fundamentals/code-analysis/style-rules/ide0005) [/source/playground/ClusterStress/ClusterStress.csproj::TargetFramework=net6.0]
0 Warning(s)
3 Error(s)

Time Elapsed 00:00:30.22
The command '/bin/sh -c dotnet build -c Release' returned a non-zero code: 1

System: Debian 12
Kernel 6.2.16 x86_64 GNU/Linux

Docker version 26.0.0, build 2ae903e

IMHO docker build should be independent from host OS version.

Thanks for open-sourcing this project, this is awesome news, tempted to give it a shot.

Is there any plan to open-source a Kubernetes operator / utils to ease the deployment of Garnet, especially in cluster mode ?

docker-compose.yml:

services:
  garnet:
    image: yzy613/garnet:latest
    # image: altall/garnet:v1.0.0
    container_name: garnet
    command:
      /app/GarnetServer.exe --config-import-path /data/config/garnet.json

garnet | _________
garnet | /||_||\ Garnet 1.0.1 64 bit; standalone mode
garnet | '. \ / .' Port: 3278
garnet | '.\ /.' https://aka.ms/GetGarnet
garnet | '.'
garnet |
garnet | 08::40::57 info: GarnetServer[0] Garnet 1.0.1 64 bit; standalone mode; Port: 3278
garnet | 08::40::57 info: ArgParser[0] Configuration import from embedded resource succeeded. Path: defaults.conf.
garnet | 08::40::57 warn: ArgParser[0] The following command line arguments were not parsed: /app/GarnetServer.exe. Please check the syntax of your command. For detailed usage information run with --help.
garnet | 08::40::57 info: ArgParser[0] Configuration import from local machine succeeded. Path: /data/config/garnet.json.

I'm sure you guys already benchmarked against Redis and maybe other caching solutions.
Do you have some numbers? 😸

Are there any plans to release precompiled and signed binaries to simplify the process of hosting a standalone Garnet based cache?

The implementation of Garnet Client currently seems to be limited.

I have confirmed that with ExecuteForMemoryResult and ExecuteForMemoryResultArray, most commands can be executed.
However, MULTI/EXEC transactions require sending multiple commands at once, and it doesn't appear that there is currently a method for executing raw commands for that purpose.

How about prioritizing the provision of an client API for command queueing?

I did run just redis-benchmark -p 3278 after few seconds errors like

07::14::50 crit: Session[0] [192.168.50.212:3278] [127.0.0.1:60714] [0170180D] ProcessMessages threw exception System.NullReferenceException: Object reference not set to an instance of an object.    at Garnet.server.StorageSession.RMWObjectStoreOperation[TObjectContext](Byte[] key, ArgSlice input, ObjectOutputHeader& output, TObjectContext& objectStoreContext) in /build/source/libs/server/Storage/Session/ObjectStore/Common.cs:line 25    at Garnet.server.StorageSession.ListPush[TObjectContext](Byte[] key, ArgSlice input, ObjectOutputHeader& output, TObjectContext& objectStoreContext) in /build/source/libs/server/Storage/Session/ObjectStore/ListOps.cs:line 321    at Garnet.server.GarnetApi`2.ListLeftPush(Byte[] key, ArgSlice input, ObjectOutputHeader& output) in /build/source/libs/server/API/GarnetApiObjectCommands.cs:line 171    at Garnet.server.RespServerSession.ListPush[TGarnetApi](Int32 count, Byte* ptr, ListOperation lop, TGarnetApi& storageApi) in /build/source/libs/server/Resp/Objects/ListCommands.cs:line 75    at Garnet.server.RespServerSession.ProcessArrayCommands[TGarnetApi](TGarnetApi& storageApi) in /build/source/libs/server/Resp/RespServerSession.cs:line 458    at Garnet.server.RespServerSession.ProcessBasicCommands[TGarnetApi](Byte* ptr, RespCommand cmd, TGarnetApi& storageApi) in /build/source/libs/server/Resp/RespServerSession.cs:line 327    at Garnet.server.RespServerSession.ProcessMessages() in /build/source/libs/server/Resp/RespServerSession.cs:line 273    at Garnet.server.RespServerSession.TryConsumeMessages(Byte* reqBuffer, Int32 bytesReceived) in /build/source/libs/server/Resp/RespServerSession.cs:line 199

popped up. Running several times redis-benchmark does not work as the client disconnects with: Error: Server closed the connection

I've noticed that there are some codes or variables that haven't been used. Is this by design, or might it have been an oversight?

The objectLogDevice ：

The newCurrentAddress ：

I understand that this is an edge case and it's not recommended to do this generally, however I noticed that there's significant memory usage growth when running KEYS * multiple times in a row, even if the cache is empty.

I'm able to reproduce in version 1.0.1, compiled in VS2022 in Release/Any CPU mode and using the .NET 8 version.

I then use the memurai-cli client and type KEYS * a few times. The memory usage will grow way beyond the memory limits that I've set in my garnet.conf file (see attached).

I've narrowed it down to TsavoriteBase.cs line 113 - whenever it calls internal void Initialize(int version, long size, int sector_size) it's calling it twice, allocating ~500MB memory each time, so each time I call KEYS *, memory grows by ~1.1GB.

However, I do notice that if I do this enough times, the GC will eventually free the memory so that's not too bad.

Just wondering if there's any way to restrict the amount of memory being allocated? Or even why the KEYS command needs to allocate that much memory in the first place. It also seems like the allocator is ignoring all of the memory settings (I tried MemorySize, IndexSize, IndexMaxSize, ObjectStoreTotalMemorySize settings to no avail).

Not sure what the fix would be, if there even needs to be one.

Thanks in advance,

Config file:
garnet.conf.txt

Line 113 where it allocates ~500MB:

Memory Usage

Hi,
I ran into this project and I'm really curious. I want to go ahead and try Garnet as an alternative to Redis. However, my projects at this time are configured to use DAPR. The State Store for DAPR is configured to use Redis as an underlying storage mechanism. Can I easily swap Redis out with Garnet?

Here's your opportunity to implement a couple of RESP commands in Garnet!

SDIFF

Syntax:
SDIFF key [key ...]

Command description:
Returns the members of the set resulting from the difference between the first set and all the successive sets.

For example:

key1 = {a,b,c,d}
key2 = {c}
key3 = {a,c,e}
SDIFF key1 key2 key3 = {b,d}

Keys that do not exist are considered to be empty sets.

Response:

Array reply: a list with members of the resulting set.

SDIFFSTORE

Syntax:
SDIFFSTORE destination key [key ...]

Command description:
This command is equal to SDIFF, but instead of returning the resulting set, it is stored in destination.

If destination already exists, it is overwritten.

Response:

Integer reply: the number of elements in the resulting set.

How to approach this task:

1. Add the new command info to the setCommandsInfoMap in the RespCommandsInfo class (Garnet.server/Resp/RespCommandsInfo.cs)
2. Add the new command string to the returned HashSet in the RespInfo.GetCommands method (Garnet.server/Resp/RespInfo.cs)
3. Add the appropriate fast parsing logic for the new command in the RespCommand.FastParseArrayCommand method (use other commands as reference) (Garnet.server/Resp/RespCommand.cs)
4. Implement the wrapper method to the storage layer in StorageSession (Garnet.server/Storage/Session/ObjectStore/SetOps.cs)
5. Define a new method in IGarnetAPI and implement it in GarnetApiObjectCommands, calling the method you have implemented in step #4 (Garnet.server/API/IGarnetAPI.cs, Garnet.server/API/GarnetApiObjectCommands.cs)
6. Add a new method to the RespServerSession class which will call the storage API and return the appropriate response (Garnet.server/Resp/Objects/SetCommands.cs)
7. Add a new method to SetObjImpl which will perform the required operation on the hash object (Garnet.server/Objects/Set/SetObjectImpl.cs)
8. Add a new enum value to SetOperation and add the appropriate case to the switch in SetObject.Operate, in which you will call the method defined in step #8 (Garnet.server/Objects/Set/SetObject.cs)
9. Add an appropriate case to the switch in TransactionManager.SetObjectKeys (Garnet.server/Transaction/TxnKeyManager.cs)
10. Add tests for the new command in the RespSetTests class (Garnet.test/RespSetTests.cs)

Tip: First add a simple test that calls the new command and use it to debug as you develop, it will make your life much easier!

If you have any questions, the Garnet team is here to help!

I followed the guide from "getting started" (https://microsoft.github.io/garnet/docs/getting-started). When I run "dotnet build -c Release", I got "Build FAILED." prompt as following messages.

OS: AlmaLinux release 9.3
Kernel: Linux devhost1 5.14.0-362.18.1.el9_3.x86_64

 dotnet build -c Release

MSBuild version 17.8.5+b5265ef37 for .NET
  Determining projects to restore...
  Restored /home/mike/src/garnet/libs/common/Garnet.common.csproj (in 573 ms).
  Restored /home/mike/src/garnet/libs/storage/Tsavorite/cs/src/core/Tsavorite.core.csproj (in 573 ms).
  Restored /home/mike/src/garnet/playground/Bitmap/Bitmap.csproj (in 573 ms).
  Restored /home/mike/src/garnet/playground/GarnetClientStress/GarnetClientStress.csproj (in 573 ms).
  Restored /home/mike/src/garnet/libs/client/Garnet.client.csproj (in 573 ms).
  Restored /home/mike/src/garnet/benchmark/Resp.benchmark/Resp.benchmark.csproj (in 573 ms).
  Restored /home/mike/src/garnet/libs/cluster/Garnet.cluster.csproj (in 573 ms).
  Restored /home/mike/src/garnet/libs/server/Garnet.server.csproj (in 573 ms).
  Restored /home/mike/src/garnet/samples/GarnetClientSample/GarnetClientSample.csproj (in 488 ms).
  Restored /home/mike/src/garnet/libs/storage/Tsavorite/cs/src/devices/AzureStorageDevice/Tsavorite.devices.AzureStorageDevice.csproj (in 573 ms).
  Restored /home/mike/src/garnet/main/GarnetServer/GarnetServer.csproj (in 573 ms).
  Restored /home/mike/src/garnet/libs/host/Garnet.host.csproj (in 573 ms).
  Restored /home/mike/src/garnet/playground/Embedded.perftest/Embedded.perftest.csproj (in 578 ms).
  Restored /home/mike/src/garnet/playground/TstRunner/TstRunner.csproj (in 592 ms).
  Restored /home/mike/src/garnet/samples/MetricsMonitor/MetricsMonitor.csproj (in 16 ms).
  Restored /home/mike/src/garnet/test/Garnet.test.cluster/Garnet.test.cluster.csproj (in 614 ms).
  Restored /home/mike/src/garnet/playground/ClusterStress/ClusterStress.csproj (in 40 ms).
  Restored /home/mike/src/garnet/test/Garnet.test/Garnet.test.csproj (in 619 ms).
  1 of 19 projects are up-to-date for restore.
/usr/lib64/dotnet/sdk/8.0.103/Roslyn/Microsoft.CSharp.Core.targets(84,5): error : Unhandled exception. Interop+Crypto+OpenSslCryptographicException: error:03000098:digital envelope routines::invalid digest [/home/mike/src/garnet/metrics/HdrHistogram/HdrHistogram.csproj::TargetFramework=net6.0]
/usr/lib64/dotnet/sdk/8.0.103/Roslyn/Microsoft.CSharp.Core.targets(84,5): error :    at Interop.Crypto.RsaSignHash(SafeEvpPKeyHandle pkey, RSASignaturePaddingMode paddingMode, IntPtr digestAlgorithm, ReadOnlySpan`1 hash, Span`1 destination) [/home/mike/src/garnet/metrics/HdrHistogram/HdrHistogram.csproj::TargetFramework=net6.0]
/usr/lib64/dotnet/sdk/8.0.103/Roslyn/Microsoft.CSharp.Core.targets(84,5): error :    at System.Security.Cryptography.RSAOpenSsl.TrySignHash(ReadOnlySpan`1 hash, Span`1 destination, HashAlgorithmName hashAlgorithm, RSASignaturePadding padding, Boolean allocateSignature, Int32& bytesWritten, Byte[]& signature) [/home/mike/src/garnet/metrics/HdrHistogram/HdrHistogram.csproj::TargetFramework=net6.0]
/usr/lib64/dotnet/sdk/8.0.103/Roslyn/Microsoft.CSharp.Core.targets(84,5): error :    at System.Security.Cryptography.RSAOpenSsl.SignHash(Byte[] hash, HashAlgorithmName hashAlgorithm, RSASignaturePadding padding) [/home/mike/src/garnet/metrics/HdrHistog

After setting an object and letting it expire, when running the SCAN command, Garnet still lists the key, while Redis does not.

This causes issues with RedisInsight and Another Redis Desktop Manager as they will run the TYPE command on each returned item from the SCAN command. The TYPE command bug is addressed in #150, however, even with this fix, in RedisInsight, the expired keys will show as having "No limit", and it'll show an error saying that the key with that name does not exist.

Using version 1.0.1, compiled in VS2022 in Release/Any CPU mode and using the .NET 8 version.

Garnet:

Redis:

RedisInsight error:

The PING command is one of only a few that are permitted once a connection has entered subscription mode; however, it has a different semantic:

If the client is subscribed to a channel or a pattern, it will instead return a multi-bulk with a "pong" in the first position and an empty bulk in the second position, unless an argument is provided in which case it returns a copy of the argument.

A typical reply might therefore be:

*2
$4
pong
$0

However, Garnet issues instead:

+PONG

To repro, try (looking in the obvious directory):

using StackExchange.Redis;
using StackExchange.Redis.Configuration;

var options = ConfigurationOptions.Parse("127.0.0.1:3278");
LoggingTunnel.LogToDirectory(options, @"c:\Code\RedisLog");
using var muxer = ConnectionMultiplexer.Connect(options);
var sub = muxer.GetSubscriber();
var ttl = sub.Ping();
Console.WriteLine(ttl);

This may impact clients that aren't expecting it; I think SE.Redis gracefully survives this, but I haven't checked for niche cases.

It may also be worth checking whether Garnet is responding to any other commands in subscriber mode; there are very few things it should allow while there are subscriptions assuming RESP2:

• PING (with different behavior)
• [P|S][UN]SUBSCRIBE
• RESET
• QUIT

Here's your opportunity to implement a couple of RESP commands in Garnet!

SINTER

Syntax:
SINTER key [key ...]

Command description:
Returns the members of the set resulting from the intersection of all the given sets.

For example:

key1 = {a,b,c,d}
key2 = {c}
key3 = {a,c,e}
SINTER key1 key2 key3 = {c}

Keys that do not exist are considered to be empty sets. With one of the keys being an empty set, the resulting set is also empty (since set intersection with an empty set always results in an empty set).

Response:

Array reply: a list with members of the resulting set.

SINTERSTORE

Syntax:
SINTERSTORE destination key [key ...]

Command description:
This command is equal to SINTER, but instead of returning the resulting set, it is stored in destination.

If destination already exists, it is overwritten.

Response:

Integer reply: the number of elements in the resulting set.

How to approach this task:

1. Add the new command info to the setCommandsInfoMap in the RespCommandsInfo class (Garnet.server/Resp/RespCommandsInfo.cs)
2. Add the new command string to the returned HashSet in the RespInfo.GetCommands method (Garnet.server/Resp/RespInfo.cs)
3. Add the appropriate fast parsing logic for the new command in the RespCommand.FastParseArrayCommand method (use other commands as reference) (Garnet.server/Resp/RespCommand.cs)
4. Implement the wrapper method to the storage layer in StorageSession (Garnet.server/Storage/Session/ObjectStore/SetOps.cs)
5. Define a new method in IGarnetAPI and implement it in GarnetApiObjectCommands, calling the method you have implemented in step #4 (Garnet.server/API/IGarnetAPI.cs, Garnet.server/API/GarnetApiObjectCommands.cs)
6. Add a new method to the RespServerSession class which will call the storage API and return the appropriate response (Garnet.server/Resp/Objects/SetCommands.cs)
7. Add a new method to SetObjImpl which will perform the required operation on the hash object (Garnet.server/Objects/Set/SetObjectImpl.cs)
8. Add a new enum value to SetOperation and add the appropriate case to the switch in SetObject.Operate, in which you will call the method defined in step #8 (Garnet.server/Objects/Set/SetObject.cs)
9. Add an appropriate case to the switch in TransactionManager.SetObjectKeys (Garnet.server/Transaction/TxnKeyManager.cs)
10. Add tests for the new command in the RespSetTests class (Garnet.test/RespSetTests.cs)

Tip: First add a simple test that calls the new command and use it to debug as you develop, it will make your life much easier!

If you have any questions, the Garnet team is here to help!

How to use the Command within the process?
IGarnetApi GarnetServer

There is a nodejs application that uses redis without problems, but if I switch to garnet this error will appear:

node:internal/process/promises:288
             triggerUncaughtException(err, true /* fromPromise */);
             ^

[ErrorReply: ERR unknown command]

Node.js v18.19.1

This is the client configuration

const client = redis.createClient({
     url: 'redis://127.0.0.1:3278'
});

This is garnet's log

08::57::25 dbug: GarnetServer[0] [****:3278] Accepted TCP connection from 127.0.0.1:48968
08::57::25 dbug: Session[0] [****:3278] [127.0.0.1:48968] [0014B75E] Starting RespServerSession
08::57::25 dbug: Session[0] [****:3278] [127.0.0.1:48968] [0014B75E] Disposing RespServerSession

I am wondering if there is any plan to support Redis Sentinel mode? Or are there any guidance on how to make Redis Sentinel client work with Garnet seamlessly?
Thanks

Redis supports streams. Is this feature also in roadmap?

We discussed this a bit earlier in the Garnet discord but I'm opening issue for tracking it here.

In .NET 6, the File IO in the BCL was essentially entirely rewritten, including introduction of new RandomAccess low-level API. The RandomAccess API is extremely optimized low-level offset based thread-safe IO wrapper that depending on the platform, either uses pread(v)/pwrite(v) for Unix or Overlapped IO for Windows.

We should investigate if RandomAccess based storage device implementation can be added to Tsavorite.core.

Would like to know if there're plans to release the Docker image in DockerHub?

Is Garnet provides supporting more than RAM memory capability?

I've just build garnet on my windows machine from the v1.0.0 tag.
The Command "command" only returns 3 commands, is that working as intended or did i do something wrong?

I'm opening this issue to get some input from the maintainers on the possibility of having the projects code-style enforced with .editorconfig rules.

Code-style such as naming conventions are very subjective and it would be nice if contributors could just do project wide dotnet format [--verify-no-changes] after doing their changes. This would reduce mental overhead of trying to deduce the correct naming for variables and fields (which are currently inconsistent, even across singular files in the project) and make future code reviews easier.

I'm most concerned about the inconsistency of the field and variable naming conventions and would like to suggest bumping all the dotnet_naming_rule.*.severity rules from suggestion to warning and then running dotnet format for the entire repository. This should avoid the manual work such as #84.

Couple examples of naming inconsistencies

Another thing that makes reading the code harder is the (inconsistenly) missing access and visibility modifiers from fields.

It's nice to see there is already effort to put into enforcing code-style already (see #106 etc.) but the naming would be nice to get under control early on 😄

_{I am myself big fan of the .NET Runtime code-style + file scoped namespaces (applying file scoped namespaces can be done with Visual Studio to the entire solution, but probably wise to time so that it would not cause big conflicts with inprogress PRs)}

It was found that when a port is occupied by another program, it can cause an exception to be thrown from Start() method in GarnetServer.cs. Then the dispose() method wil be executed (when the above exception was catched outside), and all the files in the working directory will be deleted automatically without any warnings or reminders by "ckptdir.Delete()". Is this a bug?

public void Dispose()
   {
       InternalDispose();

       logFactory?.Delete(new FileDescriptor { directoryName = "" });
       if (opts.CheckpointDir != opts.LogDir && !string.IsNullOrEmpty(opts.CheckpointDir))
       {
           var ckptdir = opts.DeviceFactoryCreator();
           ckptdir.Initialize(opts.CheckpointDir);
           ckptdir.Delete(new FileDescriptor { directoryName = "" });
       }
   }

in this case opts.LogDir is the default value (null ).

Can multiple DB support and "SELECT" command support be added. If you want to replace Redis, it is difficult to achieve without multi DB support. If supported, replacing Redis would be much more convenient.

This is beyond features that Redis offers; I have discussed similar with Redis, but it hasn't gone anywhere; maybe Garnet can trailblaze here?

The FIFO nature of Redis request/response means that two scenarios cause severe head-of-line blocking problems:

• explicitly blocking operations (BRPOP etc)
• anything that stalls the data - checkpoints, slow disk read, etc

Wouldn't it be nice if async completion was possible, WITHIN the existing RESP protocol?

Proposal:

Add a feature that is consistent with the pattern used by the client side caching Redis feature, composed of four parts:

1. ASYNC {ON|OFF} [REDIRECT conn] [OPTIN] - enables the async feature on a connection, optionally specifying a redirect pub/sub connection (required on RESP2, disallowed on RESP3), and the optional OPTIN that indicates whether this is on/off by default on a command-by-command basis; response: +OK
2. ASYNC {YES|NO} - (requires ASYNC ON first) optional; explicitly enable/disable async for the next operation (typically used as ASYNC YES to turn async on for an individual call, when OPTIN is enabled) - response: +OK
3. -ASYNC token new error response from the server when a specific operation has gone out-of-band, where token is the correlation identifier to match this result when available (only used when ASYNC ON is in operation, and the command is electing to use async)
4. "push" message (on the redirect connection for RESP2, on the current connection for RESP3) that is an array with 3 parts: the category identifier "async", the correlation identifier, and the RESP fragment that is the response for that message; this includes all possible RESP responses including errors, etc

When is a command eligible for async?

• ASYNC ON must have been issued, and no subsequent ASYNC OFF
• if OPTIN was specified, there must be an ASYNC YES immediately† before the message
• there must not be an ASYNC NO immediately† before the message

†=allowing for stacking with other "applies to the next command" commands, for example CLIENT CACHING

Notes:

• async responses are optional; if the response is available synchronously, the result is returned instead of -ASYNC token
• the correlation id is issued server-side and opaque; there is no planned API that allows a client to do anything with it
• the server must guarantee uniqueness of the correlation id in the scope of a single connection (conflicting identifiers on unrelated connections is explicitly permitted)
• any clients not prepared for async see this as a redis error that is clear - it won't work, but it also won't smash clients to pieces; this is directly comparable to -MOVED for cluster
• clients that are async aware can silently ignore -ERR from ASYNC commands on servers that do not implement it; the next command will simply return synchronously which is exactly what we expect - no change to sync path
• tokens do not need to be issued or stored/matched unless we go async: zero impact to sync path
• tokens must be compatible with "simple error" strings; opaque integer token makes a lot of sense (concretely: "simple error" and "simple string" values must use printable ASCII excluding \r and \n)
• choosing to use ASYNC is inherently also saying "I'm ok with this going out of FIFO"
• async does not operate for commands inside MULTI (responses will -QUEUED, including for ASYNC messages) or for WATCH / DISCARD (response will always be synchronous); the server may (but is not required to) allow async completion for EXEC
• async does not operate within Lua (responses will be synchronous)
• async does not add timeout behaviour; commands that do already have a timeout (brpop etc) would be expected to work as such, though
• the server is permitted to apply quotas or other reasons to not use an async response; for example, the server may wish to allow only N outstanding async commands per connection; it could choose to switch to sync only at that point, or could issue -ERR too many async operations
• outstanding async operations are considered "doomed" if a connection ends for any reason, and should be discarded by client and server (obviously doing any necessary book-keeping/cleanup)
• the ASYNC command will never itself give a -ASYNC response
• connection state modifier commands like ASKING, READONLY, READWRITE, SELECT are always synchronous
• there is no "order" in async; the ordered responses -ASYNC 123 and -ASYNC 456 may receive async responses in the order 456 then 123

Typical usage:

C: ASYNC ON REDIRECT 12344
S: +OK
C: GET foo
S: "adskjhsdfjkh"
C: GET bar
S: -ASYNC 3434534
C: GET blap
S: "asldkfslkfdgh"

.... (at some point, on subscription connection 12344)
S: ["async" "3434534" "askdjhasdkjsah"]

On RESP3 the REDIRECT 12344 would be omitted, and the server would respond (at some point) via "push" on the same connection.

Alternatively:

C: ASYNC ON REDIRECT 12344
S: +OK
C: ASYNC YES  # next command is allowed (not required) to give async response
S: +OK
C: GET foo
S: "adskjhsdfjkh"
C: ASYNC YES    # next command is allowed (not required) to give async response
S: +OK
C: GET bar
S: -ASYNC 3434534
C: GET blap     # this will never go ASYNC, because no ASYNC YES
S: "asldkfslkfdgh"
C: ASYNC NO
S: +OK
C: GET blap     # this will never go ASYNC, because ASYNC NO
S: "asldkfslkfdgh"

.... (at some point, on subscription connection 12344)
S: ["async" "3434534" "askdjhasdkjsah"]

I am happy to offer the time to get such an idea working with SE.Redis

Here's your opportunity to implement a RESP command in Garnet!

Syntax:
HSTRLEN key field

Command description:
Returns the string length of the value associated with field in the hash stored at key. If the key or the field do not exist, 0 is returned.

Response:
Integer Reply: the string length of the value associated with the field, or zero when the field isn't present in the hash or the key doesn't exist at all.

How to approach this task:

1. Add the new command info to the hashCommandsInfoMap in the RespCommandsInfo class (Garnet.server/Resp/RespCommandsInfo.cs)
2. Add the new command string to the returned HashSet in the RespInfo.GetCommands method (Garnet.server/Resp/RespInfo.cs)
3. Add the appropriate fast parsing logic for the new command in the RespCommand.FastParseArrayCommand method (use other commands as reference) (Garnet.server/Resp/RespCommand.cs)
4. Implement the wrapper method to the storage layer in StorageSession (Garnet.server/Storage/Session/ObjectStore/HashOps.cs)
5. Define a new method in IGarnetAPI and implement it in GarnetApiObjectCommands, calling the method you have implemented in step #4 (Garnet.server/API/IGarnetAPI.cs, Garnet.server/API/GarnetApiObjectCommands.cs)
6. Add a new method to the RespServerSession class which will call the storage API and return the appropriate response (Garnet.server/Resp/Objects/HashCommands.cs)
7. Add a new method to HashObjImpl which will perform the required operation on the hash object (Garnet.server/Objects/Hash/HashObjectImpl.cs)
8. Add a new enum value to HashOperation and add the appropriate case to the switch in HashObject.Operate, in which you will call the method defined in step #8 (Garnet.server/Objects/Hash/HashObject.cs)
9. Add an appropriate case to the switch in TransactionManager.HashObjectKeys (Garnet.server/Transaction/TxnKeyManager.cs)
10. Add tests for the new command in the RespHashTests class (Garnet.test/RespHashTests.cs)

Tip: First add a simple test that calls the new command and use it to debug as you develop, it will make your life much easier!

If you have any questions, the Garnet team is here to help!