Redis Clustering in Docker Swarm, Silo's wont find each other about orleans.redis

[NangiDev]

If I kill the node the silo is on it will automatically spawn a new silo on another node. What happens in the membership table is that the old silo will still be in status 3 and the new silo will try to connect but never go past status 2. Until I manually delete the old silo from the membership table and lets the new reboot.

I'm suspecting it's the communication between silos that doesn't work and that's why a second silo will never be "healthy" will the old one that got spun up alone will just assume its the new that is wrong.

What can I do to debug whats wrong with the communication between silos? I have exec into the containers and ping'ed their ips shown in the membership table. It seems fine.

[NangiDev]

I think I've solved it. I changed to define AdvertisedIp instead of hostname when configure endpoint,
Seems to have done the trick.

This
ConfigureEndpoints(advertisedIP: IPAddress.Parse(EnvVar.LocalAddress), siloPort: EnvVar.siloPort, gatewayPor t: EnvVar.gatewayPort, listenOnAnyHostAddress: true)

Instead of this
.ConfigureEndpoints(hostname: EnvVar.Hostname, siloPort: EnvVar.siloPort, gatewayPort: EnvVar.gatewayPort, listenOnAnyHostAddress: true)

EDIT:

I recommend setting siloname aswell otherwise you will get wierd behaviour with dead silos still having status 3 in Redis membership table.

This works for me:

            builder
            .Configure<SiloOptions>(options =>
            {
                options.SiloName = EnvVar.Hostname;
            })
            .ConfigureEndpoints(advertisedIP: IPAddress.Parse(EnvVar.LocalAddress), siloPort: EnvVar.siloPort, gatewayPort: EnvVar.gatewayPort, listenOnAnyHostAddress: true)
            .AddRedisGrainStorageAsDefault(optionsBuilder => optionsBuilder.Configure(options =>
            {
                options.ConnectionString = "storage:6379";
                options.UseJson = true;
                options.DatabaseNumber = 1;
            }))
            .AddRedisGrainDirectory(
                "redis-directory-1",
                options => options.ConfigurationOptions = new ConfigurationOptions
                {
                    EndPoints = { { "storage:6379" } },
                    DefaultDatabase = 2
                })
            .UseRedisClustering(options =>
            {
                options.ConnectionString = "storage:6379";
                options.Database = 0;
            });