All keys deleted after Redis server rebooted unexpectedly about orleans.redis

[SebastianStehle]

Yes, you are right. I should have read it more carefully. I think your observations are correct:

1. Not every RedisServerException is a migration problem.
2. Transactions are useless here.
3. Delete should not be used.
4. For migrations the key should just be overriden.

[jarkkojasberg]

Also lua script for write command should be loaded again to Redis server when RedisServerException with NOSCRIPT message is received.

The script cache is cleared when Redis server restarts. Lua script for write operation is initialized only in RedisGrainStorage.Init function. This means that application using this library is throwing RedisServerException with NOSCRIPT message until it is also restarted, because RedisGrainStorage.Init is only called when application starts.

Eval command documentation mentions that script cache is cleared in restart:
https://redis.io/commands/eval

Article that explains this issue:
http://www.wjwh.eu/posts/2020-06-15-redis-eval-trick.html

[jarkkojasberg]

Here is a graph that shows redis key count and how keys are deleted gradually after reboot.

[SebastianStehle]

I think this is the default for redis. You have to turn on disk persistence

[jarkkojasberg]

We have configured RDB persistence for Redis server and also tested that data is restored from snapshot after reboot. If the persistence didn't work we should have lost all keys immediately when server was rebooted. Now keys were deleted gradually after reboot. I'm quite sure that the migration code in RedisGrainStorage.WriteStateAsync function is deleting keys from Redis server.

[ReubenBond]

Are you able to take a look at fixing this, @jarkkojasberg?

[jarkkojasberg]

@ReubenBond I have been quite busy lately, I can't promise fixing this anytime soon.

[f-i-x-7]

Recently our team encountered same issue with permanent receival of exception as @jarkkojasberg mentioned in 1st post until Orleans silo was restarted. Most probably the reason was Redis server restart too (I didn't perform a deep investigation). We have .RDB persistence enabled too. Fortunately, this was a test environment.
May be I will be able to prepare suggested fix. But I did not see any contributing guidelines in this repo. If there are any, please let me know.

[f-i-x-7]

Hello again,

As our team uses Orleans in production and we possibly will migrate some grains from ADO.NET/EF storage to Redis next year, we want to be sure in stable work of RedisGrainStorage. So one of my colleagues is going to fix this bug.

After discussion with him I've checked the code in RedisGrainStorage more carefully. It seems that migration logic from WriteStateAsync() can be safely removed because of 2 reasons:

1. Every grain at first reads state from storage. So if migration needed, it would be already performed duringReadStateAsync() call.
2. Migration logic in WriteStateAsync(), if executed, actually would fail. It uses the same Lua script as regular WriteStateAsync() operation. This script uses HGET command to retrieve Palyload and ETag. But for old grain storage format HGET would throw (something like "(error) ERR Error running script (call to f_6b1bf486c81ceb7edf3c093f4c48582e38c0e791): ERR Operation against a key holding the wrong kind of value").

So, as part of the bugfix, we are going just to remove migration logic from WriteStateAsync().

@ReubenBond @SebastianStehle are you OK with that?

[f-i-x-7]

It seems that migration logic from WriteStateAsync() can be safely removed

Well, actually after deletion of migration logic from WriteStateAsync() one test failed. So I decided to fix migration logic in WriteStateAsync() - do not use Lua script but execute same commands within transaction as migration in ReadStateAsync() does.

BTW:

Yes, you are right. I should have read it more carefully. I think your observations are correct:

1. Not every RedisServerException is a migration problem.
2. Transactions are useless here.
3. Delete should not be used.
4. For migrations the key should just be overriden.

Actually delete MUST be used, because string key in Redis cannot be overwritten by hash. So IMHO it is OK to stick with transactions as of now (other option is to switch to another Lua scripts, but it will bring additional complexity related with handling of NOSCRIPT errors, etc.).

What about 1st point - "Not every RedisServerException is a migration problem." - I've added some checks to decide whether migration is required.