orleanscontrib / orleans.redis

Redis support packages for Orleans

Orleans Redis Providers

1.5.x branch 2.x.x branch

Orleans is a framework that provides a straight-forward approach to building distributed high-scale computing applications, without the need to learn and apply complex concurrency or other scaling patterns.

Redis is an open source (BSD licensed), in-memory data structure store, used as a database, cache and message broker.

StackExchange.Redis library underneath.

Orleans.Persistence.Redis

Attention: Version 7.x is NOT binary compatible with previous versions.
Configuration and serialization have changed to align with serialization changes in the Orleans framework 7.x

Installation

PS> Install-Package Orleans.Persistence.Redis -prerelease

Usage

Configure your Orleans silos

var silo = new SiloHostBuilder()
    .AddRedisGrainStorage("Redis", optionsBuilder => optionsBuilder.Configure(options =>
    {
        options.DataConnectionString = "localhost:6379"; // This is the default
        options.DatabaseNumber = 1;
    }))
    .Build();
await silo.StartAsync();

Decorate your grain classes with the StorageProvider attribute or use DI with the PersistentState attribute as needed.

[StorageProvider(ProviderName = "Redis")]
public class SomeGrain : Grain<SomeGrainState>, ISomeGrain

public class SomeGrain2 : Grain, ISomeGrain2 {
   SomeGrain2(
       [PersistentState("state", "Redis")] IPersistentState<SomeGrain2State> state
   )
}

These settings will enable the Redis cache to act as the store for grains that have persistent state.

Configuration

• DataConnectionString="..." (required) the connection string to your redis database (i.e. localhost:6379, is passed directly to StackExchange.Redis)
• DatabaseNumber=1 (optional) the number of the redis database to connect to. Defaults to 0.

Orleans.Clustering.Redis

Orleans clustering provider for Redis

Orleans.Clustering.Redis enables Orleans applications to use Redis as a backend for cluster membership.

Redis is a straight key/value store. Membership data is stored as a hash.

If you want to quickly test it, clone this repo and go to the samples directory for instructions on how to run a sample cluster.

Installation

Installation is performed via NuGet

From Package Manager:

Install-Package Orleans.Clustering.Redis

.Net CLI:

dotnet add package Orleans.Clustering.Redis

Configuration

A functional Redis database is required for this provider to work.

Silo

Tell Orleans runtime that we are going to use Redis as our Cluster Membership Provider:

var silo = new SiloHostBuilder()
        ...
        .UseRedisClustering(opt =>
        {
            opt.ConnectionString = "host:port";
            opt.Database = 0;
        })
        ...
        .Build();

ConnectionString tells the connector where to find the Redis database.

Database is an integer which tells the membership table which database to get after connecting to the Redis service.

More information on connection string configuration can be found at on the StackExchange.Redis driver site (https://stackexchange.github.io/StackExchange.Redis/Configuration.html).

Client

Now that our silo is up and running, the Orleans client needs to connect to the Redis database to look for Orleans gateways.

var client = new ClientBuilder()
        ...
        .UseRedisClustering(opt =>
        {
            opt.ConnectionString = "host:port";
            opt.Database = 0;
        })
        ...
        .Build();

At the moment the gateway list is provided by the underlying membership provider directly.

License

This project is licensed under the MIT license.

orleans.redis's Issues

Redis Streaming

What about implementing streaming adapter using redis PUB\SUB ?

I have found https://github.com/berdon/Orleans.Providers.Redis but it's not in OrleansContrib repository and have questions with using ConcurrentQueue and pool of channels instead of channel per subscription

Grain multiple states

Hi looks like U are not supporting multiple states inside grains.

Like:

[PersistentState("RealtimeInteractions", OrleansConstants.SessionServicePersistenceStoreName)] IPersistentState interactions;

Orleans 8.0 compatibility

AddRedisGrainStorage exception:

System.TypeLoadException: 'Could not load type 'Orleans.Runtime.KeyedServiceExtensions' from assembly 'Orleans.Core, Version=8.0.0.0, Culture=neutral, PublicKeyToken=null'.'

Unable to resolve service for type Orleans.Runtime.IPersistentState

I have aspnet core web api project, so I wanted to use Redis provider for clustering and storage.

Program.cs was like this

int db = 2;
string redisConnStr = "localhost:6379";

builder.Host.UseOrleans((host, builder) =>
{
    builder.AddRedisGrainStorage("RedisGrainStorage", optionsBuilder => optionsBuilder.Configure(options =>
    {
        options.ConnectionString = redisConnStr;
        options.UseJson = true;
        options.DatabaseNumber = db;
    }));

    builder.UseRedisClustering(options =>
    {
        options.ConnectionString = redisConnStr;
        options.Database = db;
    });

    builder.Configure<ClusterOptions>(options =>
    {
        options.ClusterId = "_basket_cluster";
        options.ServiceId = "_basket_service";
    });

    builder.Configure<EndpointOptions>(options =>
    {
        options.AdvertisedIPAddress = IPAddress.Loopback;
        options.GatewayListeningEndpoint = new IPEndPoint(IPAddress.Any, EndpointOptions.DEFAULT_GATEWAY_PORT);
        options.SiloListeningEndpoint = new IPEndPoint(IPAddress.Any, EndpointOptions.DEFAULT_SILO_PORT);
    });

    builder.ConfigureLogging(logging => logging.AddConsole());
});

builder.Services.AddTransient<IBasketActor, BasketActor>();

And I have a grain with name is BasketActor
Then when I want to persistence my state with IPersistence

[StorageProvider(ProviderName = "RedisGrainStorage")]
    public class BasketActor : Grain, IBasketActor
    {
        public BasketActor(
            [PersistentState("basket", "RedisGrainStorage")] IPersistentState<BasketState> basket
            )
        {
        }
    }

And I have a grain with name is BasketActor
Then when I want to persistence my state with IPersistence
I am getting error this;

** Error while validating the service descriptor 'ServiceType:Basket.Actors.IBasketActor Lifetime: Transient ImplementationType: Basket.Actors.BasketActor': Unable to resolve service for type 'Orleans.Runtime.IPersistentState`1Basket.Actors.BasketState]' while attempting to activate 'Basket.Actors.BasketActor'.**

I've mentioned before in discord

Thanks.

[Question] intention of redis key format (ServiceId/ClusterId)

when i start a client use redis clustering, i can't got cluster , because no key [ServiceId/ClusterId] found in redis.

e.g.

Service 1

hostBuilder.UseOrleans((context, builder) => {
    builder.Configure<ClusterOptions>(opt=>{
        opt.ClusterId = "dev";
        opt.ServiceId = "service1";
    });
});

Service 2

hostBuilder.UseOrleans((context, builder) => {
    builder.Configure<ClusterOptions>(opt=>{
        opt.ClusterId = "dev";
        opt.ServiceId = "service2";
    });
});

Client

new ClientBuilder()
.Configure<ClusterOptions>(opt=>{
        opt.ClusterId = "dev";
        opt.ServiceId = "client1";
})
.UseRedisClustering(...)
.Build()
.Connect(...)

other cluster implements just use ClusterId
AdoNet

Consul

ZooKeeper

RedisGrainStorage.ReadStateAsync() should use Array.Length instead of Linq.Enumerable.Count()

There is a possibility for micro perf improvement by replacing this line:

Orleans.Redis/src/Orleans.Persistence.Redis/RedisGrainStorage.cs

Line 124 in 1de3c47

if (hashEntries.Count() == 2)

to this:

if (hashEntries.Length == 2)

hashEntries is an array, so this is technically possible, and it should benefit by eliminating virtual call to ICollection<T>.Count that is called by Linq.Enumerable.Count() under the hood (in theory it is possible that due to several inlines made by JIT a possibility for guarded devirtualization can occur, but I think it is better to reduce JIT work anyway and just access Length directly).

This is very simple change, so if you are interested in it, I could submit a PR.

[Question] Can it be used on ISiloBuilder?

Hi, can this provider be used in the scenario of combined client+host? I am using this code

await Host.CreateDefaultBuilder(args)
    .UseOrleans(builder =>
    {
        builder.UseLocalhostClustering();
        builder.AddMemoryGrainStorageAsDefault();
        builder.AddSimpleMessageStreamProvider("SMS");
        builder.AddMemoryGrainStorage("PubSubStore");
    })
    .ConfigureWebHostDefaults(webBuilder =>
    {
        webBuilder.UseStartup<Startup>();
    })
    .RunConsoleAsync(); 

but ISiloBuider is not recognising the extension to add Redis (nuget installed);

FLUSHALL required between running different code

I am working Orleans samples here https://github.com/dodyg/practical-aspnetcore/tree/net6.0/projects/orleans that uses this AddRedisGrainStorage. The thing is I have to FLUSHALL in redis-cli between samples because serializations errors. How can I prevent this from happening? Is there any specific key or namespace I can add?

Orleans 4.0 compatibility

The new preview for Orleans is available. It would be great to get a compatible Redis package

Does not work with Co-Hosting

When we use orleans co-hosting with ASP.NET we do not have ISiloHostBuild and have only ISiloBuilder.
So we can't use extensions like AddRedisGrainStorage.

GrainReference not deleted from redis

Hi, I am working to implement Orleans sagas in our system and I find that the GrainReference does not get removed from redis once the saga is completed.

This is my configuration:

.UseOrleans((ctx, orleansBuilder) =>
            {
                orleansBuilder
                    .UseSagas(typeof(TSagaAssembly).Assembly)
                    .ConfigureServices(services => services.Scan(scan => scan
                        .FromAssemblyOf<TSagaAssembly>()
                        .AddClasses()
                        .AsSelfWithInterfaces()
                    ))
                    .UseRedisReminderService(opt => opt.ConnectionString = redisAddress)
                    .AddRedisGrainStorageAsDefault(options =>
                    {
                        options.ConnectionString = redisAddress;
                        options.DeleteOnClear = true; //<- here I assume this will do it.
                        
                    })
                    .UseRedisClustering(options =>
                    {
                        options.ConnectionString = redisAddress;
                    });

                orleansBuilder.Configure<EndpointOptions>(options =>
                {
                    // Port to use for Silo-to-Silo
                    options.SiloPort = 11111;
                    // Port to use for the gateway
                    options.GatewayPort = 30000;
                    // IP Address to advertise in the cluster
                    options.AdvertisedIPAddress = IPAddress.Parse("127.0.0.1"); //TODO: it should be host ip
                    // The socket used for silo-to-silo will bind to this endpoint
                    options.GatewayListeningEndpoint = new IPEndPoint(IPAddress.Any, 40000);
                    // The socket used by the gateway will bind to this endpoint
                    options.SiloListeningEndpoint = new IPEndPoint(IPAddress.Any, 50000);
                });
                orleansBuilder.Configure<ClusterOptions>(options =>
                {
                    options.ServiceId = $"{redisGrainStorageName}-service";
                    options.ClusterId = $"{redisGrainStorageName}-cluster";
                });
            })

I assumed that DeleteOnClear will clear up the record but seems like it doesn't.
As I am still able to see it in Redis.

Also, I am using the version 3.1.1 (i need it to be compatible with netcore3.1);

there is no error or anything additional, is there any way to remove it? Thanks.

edit: updated the version to the correct one.

Orleans.Persistence.Redis Matches the Orleans3.X version test package

If there are two types of persistence in the same Grain, an error will be reported. Can you concatenate the field name into the Key? This can solve many problems

Could not find a membership entry for this silo

Hello. I need help. thx！

Run client and silo in the same program，and after a few seconds, log an error

[ERR Orleans.Runtime.MembershipService.LocalSiloHealthMonitor] Could not find a membership entry for this silo
[WRN Orleans.Runtime.MembershipService.LocalSiloHealthMonitor] Self-monitoring determined that local health is degraded. Degradation score is 8/8 (lower is better). Complaints: Could not find a membership entry for this silo

and after a period of time，log an error

[ERR Orleans.Runtime.MembershipService.MembershipAgent] Failed to update table entry for this silo, will retry shortly: Orleans.Clustering.Redis.RedisClusteringException: Could not find a value for the key S172.17.48.1:30000:353557710
   at Orleans.Clustering.Redis.RedisMembershipTable.UpdateIAmAlive(MembershipEntry entry)
   at Orleans.Runtime.MembershipService.MembershipTableManager.UpdateIAmAlive()
   at Orleans.Runtime.MembershipService.MembershipAgent.UpdateIAmAlive()

No such problem with UseLocalhostClustering. This repository example also has this problem.
Thank you

Redis Clustering in Docker Swarm, Silo's wont find each other

I'm having problem understand why my setup does not work.
I'm running a Redis with the membership table and I can see the silos register, but it only works with one silo. As soon as I add a second silo it never goes past 'Status:2'. It just reboots.

private static async Task<int> RunMainAsync()
{
    try
    {
        var host = new SiloHostBuilder()
            .ConfigureEndpoints(hostname: EnvVar.Hostname, siloPort: EnvVar.siloPort, gatewayPort: EnvVar.gatewayPort, listenOnAnyHostAddress: true)
            .AddRedisGrainStorageAsDefault(optionsBuilder => optionsBuilder.Configure(options =>
            {
                options.ConnectionString = "storage:6379";
                options.UseJson = true;
                options.DatabaseNumber = 1;
            }))
            .AddRedisGrainDirectory(
                "redis-directory-1",
                options => options.ConfigurationOptions = new ConfigurationOptions
                {
                    EndPoints = { { "storage:6379" } },
                    DefaultDatabase = 2
                })
            .UseRedisClustering(options =>
            {
                options.ConnectionString = "storage:6379";
                options.Database = 0;
            })
            .Configure<ClusterOptions>(options =>
            {
                options.ClusterId = EnvVar.ClusterId;
                options.ServiceId = EnvVar.ServiceId;
            })
            .ConfigureApplicationParts(parts =>
            {
                parts.AddFromApplicationBaseDirectory();
            })
            .ConfigureLogging(logging =>
            {
                logging.AddConsole();
            })
            .EnableDashboard()
        .Build();

        await host.StartAsync();

        await host.Stopped;
        Console.WriteLine("Stopping Silo host...");
        await host.StopAsync();
    }
    catch (Exception ex)
    {
        Console.WriteLine(ex);
        return 1;
    }
    return 0;
}

This us my docker-compose.yaml

version: "3.8"

networks:
        orleans_swarm:
                driver: overlay

services:
        storage:
                image: redis:6.2.5
                ports:
                        - 6379:6379
                networks:
                        - orleans_swarm

        silo:
                image: localhost:5000/silo:latest
                deploy:
                        replicas: 1
                ports:
                        - 8080:8080 # Dashboard port
                        - 50051:50051 # Grpc port
                        - 20025:20025 # Gateway port
                        - 30025:30025 # Internal Silo port
                environment:
                        - GATEWAY_PORT=20025
                        - SILO_PORT=30025
                        - GRPC_PORT=50051
                        - USE_ORLEANS_DASHBOARD=true
                        - DASHBOARD_USERNAME=admin
                        - DASHBOARD_PASSWORD=nimda
                        - CLUSTER_ID=clusterId
                        - SERVICE_ID=serviceId
                networks:
                        - orleans_swarm
                depends_on:
                        - storage

And the error in the logs:

orleans_silo.2.i02fuoaksvrl@manager    | fail: Orleans.Messaging[100071]
orleans_silo.2.i02fuoaksvrl@manager    |       Failed to address message Request S10.0.0.109:30025:370887803*stg/TypeManager/00000011@S00000011->*grn/Orleans.Runtime.Versions.VersionStoreGrain/0+clusterId InvokeMethodRequest Orleans.Runtime.Versions.IVersionStoreGrain:GetSelectorStrategies #200 from activation [SystemTarget: S10.0.0.109:30025:370887803*stg/TypeManager/00000011@S00000011]
orleans_silo.2.i02fuoaksvrl@manager    |       Orleans.Runtime.OrleansException: TypeCode $331003096 not supported in the cluster
orleans_silo.2.i02fuoaksvrl@manager    |          at Orleans.Runtime.Catalog.GetCompatibleSilos(PlacementTarget target)
orleans_silo.2.i02fuoaksvrl@manager    |          at Orleans.Runtime.Placement.RandomPlacementDirector.OnAddActivation(PlacementStrategy strategy, PlacementTarget target, IPlacementContext context)
orleans_silo.2.i02fuoaksvrl@manager    |          at Orleans.Runtime.Placement.PlacementDirectorsManager.AddActivation(PlacementTarget target, IPlacementRuntime context, PlacementStrategy strategy)
orleans_silo.2.i02fuoaksvrl@manager    |          at Orleans.Runtime.Placement.PlacementDirectorsManager.SelectOrAddActivation(ActivationAddress sendingAddress, PlacementTarget targetGrain, IPlacementRuntime context, PlacementStrategy strategy)
orleans_silo.2.i02fuoaksvrl@manager    |          at Orleans.Runtime.Dispatcher.AddressMessageAsync(Message message, PlacementTarget target, PlacementStrategy strategy, ActivationAddress targetAddress)
orleans_silo.2.i02fuoaksvrl@manager    |          at Orleans.Runtime.Dispatcher.<>c__DisplayClass39_0.<<AsyncSendMessage>g__TransportMessageAferSending|0>d.MoveNext()

I've have noticed that the ip address it tries to connect to is the ingress ip provided by Docker Swarm and not my own overlay network 'orleans_swarm'

I've no idea if that makes any difference.

My node setup is just three Ubuntu virtual machines, but for testing purposes I have drained the two nodes and only the manager is active.

Just to be clear my setup works perfect as long as I don't have more than one silo. It even works with more swarm nodes

'ISiloBuilder' does not contain a definition for 'AddRedisGrainStorage'

Although according to

Orleans.Redis/src/Orleans.Persistence.Redis/RedisPersistenceHostingExtensions.cs

Line 79 in 15b63ca

public static ISiloBuilder AddRedisGrainStorage(this ISiloBuilder builder, string name, Action<OptionsBuilder<RedisStorageOptions>> configureOptions = null)

, this should compile.

Am I using the wrong nuget source?

Cannot link redis with password

when I start silo server，and stackexchange.redis throw an exception,but the data was added to the redis.
StackExchange.Redis.RedisConnectionException: It was not possible to connect to the redis server(s). There was an authentication failure; check that passwords (or client certificates) are configured correctly. AuthenticationFailure (None, last-recv: 310) on localhost:6379/Interactive, Flushed/ComputeResult, last: ECHO, origin: SetResult, outstanding: 0, last-read: 0s ago, last-write: 0s ago, keep-alive: 60s, state: ConnectedEstablishing, mgr: 8 of 10 available, last-heartbeat: never, global: 0s ago, v: 2.1.30.38891

Orleans.Persistence.Redis storage provider doesn't respect RecordExists property

Hello.

https://github.com/OrleansContrib/Orleans.Redis/blob/main/src/Orleans.Persistence.Redis/Storage/RedisGrainStorage.cs

The IGrainState interface has the RecordExists property, but the property isn't set in any part of RedisGrainStorage's code (read, write, or clear methods). And if I try to use the RecordExists property in grain logic, I'll get the wrong information about the existence of a grain in persistent storage.

JSON deserialization of grain states does not bind GrainReferences to runtime

This provider cannot be used as PubSub store with JSON serialization format because the PubSubRendezvousGrain's PubSubGrainState contains GrainReferences on in the Consumers set (PubSubSubscriptionState.consumerReference) which does not get bound to the runtime when the state is deserialized with JsonConvert in RedisGrainStorage.ReadStateAsync:

if (_options.UseJson)
{
    grainState.State = JsonConvert.DeserializeObject(valueEntry.Value, grainState.State.GetType(), _jsonSettings);
}

The _jsonSettings does not contain a converter that would bind IAddressable instances to the runtime.

This same issue has been fixed to this library:
commit: OrleansContrib/Orleans.Providers.MongoDB@9cc9665
issue: OrleansContrib/Orleans.Providers.MongoDB#21

Also upgrading from Orleans.Persistence.Redis from 3.0.1 to 3.1 changes the default serialization format in RedisStorageOptions to JSON.

/// <summary>
/// Whether or not to use JSON for serialization.
/// </summary>
public bool UseJson { get; set; } = true;

This is a change in default behavior which I don't think should have been done in a minor version bump. This is a breaking change for usages that do not explicitly set the UseJson property in AddRedisGrainStorage.

PS. Would it be possible to add releases to this GitHub page with changelogs. Otherwise we'll have to read every line on code in the disassembled NuGet package before upgrading the library.

All keys deleted after Redis server rebooted unexpectedly

Redis server rebooted unexpectedly and all keys were deleted.

Following exception was thrown in every write operation after reboot:
Exc level 0: StackExchange.Redis.RedisServerException: NOSCRIPT No matching script. Please use EVAL. at Orleans.Persistence.RedisGrainStorage.WriteToRedisAsync(IDatabaseAsync db, Object state, String etag, String key, String newEtag) at Orleans.Persistence.RedisGrainStorage.WriteStateAsync(String grainType, GrainReference grainReference, IGrainState grainState) at Orleans.Core.StateStorageBridge1.WriteStateAsync()

It seems that logic attempts to migrate grain state every time when RedisServerException is thrown.
Is this a proper way to determine that state should be migrated?

When RedisServerException is thrown, delete and write commands are executed in the same transaction
and in this case write operation inside the transaction will fail. This exception is not caught in the code.

Redis transactions don't work the same way than in normal relational database.
Transactions are not going to rollback changes if one or more commands fails in transaction.

So reboot caused Redis server to clean all cached SHA:s for prepared scripts. Then server started to throw RedisServerException
in every write operation. This was handled as a old grain state and started migration. Delete and write commands were
executed in the same transaction. Only delete was executed successfully and all keys were deleted.

Regular null reference exceptions

It seems you should inspect all methods that you call at StackExchange.Redis. For example:

RedisValue[] values = await _db.SortedSetRangeByValueAsync(RemindersRedisKey, from, to);
if (values.Length == 0)

Wrapper could return null and you will get NRE. That's very bad.

Check here:

Orleans.Redis/src/Orleans.Reminders.Redis/RedisReminderTable.cs

Line 63 in ecc5cfa

if (values.Length == 0)