crash from failed assertion in networking.cpp about keydb

[bpo]

I assume this is intermittent?

Yes, one crash after ~1 day of uptime.

Do you have any modules loaded?

No modules

Can you give me an idea of what other commands may be in flight around the same time?

Counts by commands last executed by client:

   2 blpop
 224 brpop
   8 evalsha
  38 exec
  18 get
   1 replconf
  16 scard
   1 script
   2 set
   1 slowlog
   1 strlen
   2 xadd
1689 xreadgroup
   1 zrangebyscore

[JohnSully]

This is fixed in change aae0fdc which is now in RELEASE_5

The root cause is that shared objects such as shared.rpop aren't marked as shared. So the refcount is being incremented and decremented. This isn't a problem with single threading however in a multithreaded environment client arguments must either be local to the current thread or shared (because the refcount is modified outside the lock).

Eventually shared.rpop's refcount hits zero due to the synchronization error and is freed. At some later point malloc recycles the memory and a new object is placed there. At that point the command lookup will fail and this assert will fire.

The fix is to correctly mark these and all other similar objects as shared.
I will continue to run my stress test for a few days to further validate the fix.

[JohnSully]

I assume this is intermittent?

I see blockingPopGenericCommand in the callstack. Can you give me an idea of what other commands may be in flight around the same time? I will setup a torture test to suss it out.

Also FYI RELEASE_5 isn't officially released yet so I've been updating it silently with bugfixes (though none that I think would resolve this). At this point I will only add bug fixes not new features so I do appreciate you testing it.

[JohnSully]

Do you have any modules loaded?

I think I may have found a code path that could cause this if modules are used.

[bpo]

Same machine was showing these periodically in the logs, for a handful of fd values, not clear to me if it's related.

epoll_ctl failed: No such file or directory
2881:M 14 Jul 2019 02:33:43.201 # Error registering fd event for the new client: No such file or directory (fd=290)

Reminiscent of redis/redis#2479, but Sentinel is uninvolved and we did not see any connectivity issues on the replica (prior to the master crash that is)

[JohnSully]

It's tricky because argv[0] should be the hardcoded value "rpop" but instead is ">". Reading the code this should be impossible.

It's set to that value not far away in blockingPopGenericCommand:

rewriteClientCommandVector(c,2,
                        (where == LIST_HEAD) ? shared.lpop : shared.rpop,
                        c->argv[j]);

I have a few boxes dedicated to fuzzing, I'll focus them on brpop.