keydb crashed again, again about keydb

[JohnSully]

Were you doing anything notable at the time this occurred? Does it do this immediately upon boot?

[hellower]

It continues to occur every x*ten seconds.

[hellower]

$> pwd
/var/lib/docker/containers/21943bc683c8eda4fe93a6179d612f355e849047fea2708fde8b7e66171a6fc7

docker logs

{"log":"terminate called after throwing an instance of 'std::system_error'\n","stream":"stderr","time":"2019-05-17T19:53:32.586424774Z"}
{"log":" what(): Operation not permitted\n","stream":"stderr","time":"2019-05-17T19:53:32.586568839Z"}
{"log":"Aborted\n","stream":"stderr","time":"2019-05-17T19:53:32.58913315Z"}
{"log":"terminate called after throwing an instance of 'std::system_error'\n","stream":"stderr","time":"2019-05-17T19:54:00.635667865Z"}
{"log":" what(): Operation not permitted\n","stream":"stderr","time":"2019-05-17T19:54:00.635708836Z"}
{"log":"Aborted\n","stream":"stderr","time":"2019-05-17T19:54:00.638725202Z"}
{"log":"terminate called after throwing an instance of 'std::system_error'\n","stream":"stderr","time":"2019-05-17T19:54:33.697740965Z"}
{"log":" what(): Operation not permitted\n","stream":"stderr","time":"2019-05-17T19:54:33.697807053Z"}
{"log":"Aborted\n","stream":"stderr","time":"2019-05-17T19:54:33.699745621Z"}
{"log":"terminate called after throwing an instance of 'std::system_error'\n","stream":"stderr","time":"2019-05-17T19:55:07.756042568Z"}
{"log":" what(): Operation not permitted\n","stream":"stderr","time":"2019-05-17T19:55:07.756110882Z"}
{"log":"Aborted\n","stream":"stderr","time":"2019-05-17T19:55:07.758659871Z"}
{"log":"terminate called after throwing an instance of 'std::system_error'\n","stream":"stderr","time":"2019-05-17T19:56:17.879019025Z"}
{"log":" what(): Operation not permitted\n","stream":"stderr","time":"2019-05-17T19:56:17.879079851Z"}
{"log":"Aborted\n","stream":"stderr","time":"2019-05-17T19:56:17.881023645Z"}
{"log":"terminate called after throwing an instance of 'std::system_error'\n","stream":"stderr","time":"2019-05-17T19:56:59.956331865Z"}
{"log":" what(): Operation not permitted\n","stream":"stderr","time":"2019-05-17T19:56:59.956404987Z"}
{"log":"Aborted\n","stream":"stderr","time":"2019-05-17T19:56:59.960360608Z"}
{"log":"terminate called after throwing an instance of 'std::system_error'\n","stream":"stderr","time":"2019-05-17T19:57:00.958801893Z"}
{"log":" what(): Operation not permitted\n","stream":"stderr","time":"2019-05-17T19:57:00.958869238Z"}
{"log":"Aborted\n","stream":"stderr","time":"2019-05-17T19:57:00.960839419Z"}
{"log":"terminate called after throwing an instance of 'std::system_error'\n","stream":"stderr","time":"2019-05-17T19:57:23.999211219Z"}
{"log":" what(): Operation not permitted\n","stream":"stderr","time":"2019-05-17T19:57:23.99930523Z"}
{"log":"Aborted\n","stream":"stderr","time":"2019-05-17T19:57:24.002062239Z"}
{"log":"terminate called after throwing an instance of 'std::system_error'\n","stream":"stderr","time":"2019-05-17T19:58:01.055040319Z"}
{"log":" what(): Operation not permitted\n","stream":"stderr","time":"2019-05-17T19:58:01.055114796Z"}
{"log":"Aborted\n","stream":"stderr","time":"2019-05-17T19:58:01.057693933Z"}
{"log":"terminate called after throwing an instance of 'std::system_error'\n","stream":"stderr","time":"2019-05-17T19:58:58.161592353Z"}
{"log":" what(): Operation not permitted\n","stream":"stderr","time":"2019-05-17T19:58:58.161653248Z"}
{"log":"Aborted\n","stream":"stderr","time":"2019-05-17T19:58:58.16486516Z"}
{"log":"terminate called after throwing an instance of 'std::system_error'\n","stream":"stderr","time":"2019-05-17T19:59:21.210057989Z"}
{"log":" what(): Operation not permitted\n","stream":"stderr","time":"2019-05-17T19:59:21.210152379Z"}
{"log":"Aborted\n","stream":"stderr","time":"2019-05-17T19:59:21.212810273Z"}
{"log":"terminate called after throwing an instance of 'std::system_error'\n","stream":"stderr","time":"2019-05-17T19:59:51.264385553Z"}
{"log":" what(): Operation not permitted\n","stream":"stderr","time":"2019-05-17T19:59:51.264437015Z"}
{"log":"Aborted\n","stream":"stderr","time":"2019-05-17T19:59:51.267284123Z"}
{"log":"terminate called after throwing an instance of 'std::system_error'\n","stream":"stderr","time":"2019-05-17T20:01:42.449204297Z"}
{"log":" what(): Operation not permitted\n","stream":"stderr","time":"2019-05-17T20:01:42.449250505Z"}
{"log":"Aborted\n","stream":"stderr","time":"2019-05-17T20:01:42.451921828Z"}
{"log":"terminate called after throwing an instance of 'std::system_error'\n","stream":"stderr","time":"2019-05-17T20:04:59.784944071Z"}
{"log":" what(): Operation not permitted\n","stream":"stderr","time":"2019-05-17T20:04:59.785006734Z"}
{"log":"Aborted\n","stream":"stderr","time":"2019-05-17T20:04:59.789365865Z"}
{"log":"terminate called after throwing an instance of 'std::system_error'\n","stream":"stderr","time":"2019-05-17T20:06:34.941779181Z"}
{"log":" what(): Operation not permitted\n","stream":"stderr","time":"2019-05-17T20:06:34.94183899Z"}
{"log":"Aborted\n","stream":"stderr","time":"2019-05-17T20:06:34.944612485Z"}
.........................

/var/log/messages
..........................
May 17 19:10:53 ocean-01 sz[6112]: [root] keydb_master.conf/ZMODEM: got ZABORT
May 17 19:10:53 ocean-01 sz[6112]: [root] ZMODEM/keydb_master.conf: error occured
May 17 19:10:54 ocean-01 containerd: time="2019-05-17T19:10:54.272183644Z" level=info msg="shim reaped" id=21943bc683c8eda4fe93a6179d612f355e849047fea2708fde8b7e66171a6fc7
May 17 19:10:54 ocean-01 dockerd: time="2019-05-17T19:10:54.281921345Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
May 17 19:10:54 ocean-01 containerd: time="2019-05-17T19:10:54.422169565Z" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/21943bc683c8eda4fe93a6179d612f355e849047fea2708fde8b7e66171a6fc7/shim.sock" debug=false pid=7103
May 17 19:10:55 ocean-01 sz[7138]: [root] keydb_master.conf/ZMODEM: 71932 Bytes, 112571 BPS
May 17 19:10:55 ocean-01 sz[7138]: [root] keydb_slave.conf/ZMODEM: 71931 Bytes, 213576 BPS

[JohnSully]

It looks like it’s crashing writing out the RDB file. I will investigate.

[hellower]

Thanks!

cf.
keydb.log nothing special
......................................................................................................
KeyDB 0.9.5 (240d688/0) 64 bit
Running in standalone mode
Port: 9086
PID: 9

#Server initialized

• Loading RDB produced by version 0.9.5
• RDB age 8 seconds
• RDB memory usage when created 9.68 Mb
• DB loaded from disk: 0.000 seconds
• Before turning into a replica, using my master parameters to synthesize a cached master: I may be able to synchronize with the new master with just a partial transfer.
• Ready to accept connections
  d Thread 1 alive.
  d Thread 0 alive.
• Connecting to MASTER tnt-worker2:9086
• MASTER <-> REPLICA sync started
• Non blocking connect for SYNC fired the event.
• Master replied to PING, replication can continue...
• Partial resynchronization not possible (no cached master)
• Full resync from master: bb1356227637e1a0a929e04ae4845d91e20bb106:342790
• Discarding previously cached master state.
• MASTER <-> REPLICA sync: receiving 277 bytes from master
• MASTER <-> REPLICA sync: Keeping old data
• MASTER <-> REPLICA sync: Loading DB in memory
• Loading RDB produced by version 0.9.5
• RDB age 0 seconds
• RDB memory usage when created 9.74 Mb
• MASTER <-> REPLICA sync: Finished with success
• Replica 178.128.xxx.xxx:9086 asks for synchronization
• Full resync requested by replica 178.128.xxx.xxx:9086
• Starting BGSAVE for SYNC with target: disk
• Background saving started by pid 15
• DB saved on disk
• RDB: 0 MB of memory used by copy-on-write
• Background saving terminated with success
• Synchronization with replica xxx.xxx.xxx.xxx:9086 succeeded
  ......................................................................................................

[JohnSully]

I’m still not sure what is causing this, however I’ve updated the docker image to produce proper debugging information for uncaught exceptions.

If you can run the updated image and send the debug info it generates I should be able to resolve this fairly quickly.

Edit: We’re still updating the default images. In the meantime you can fetch the latest with:

docker pull eqalpha/keydb:x86_64_v0.9.5

[hellower]

https://www.postgresdba.com/pgexplainer/config.tgz -- master/slave configuration file
https://www.postgresdba.com/pgexplainer/master.tgz -- master log/coredump file
https://www.postgresdba.com/pgexplainer/slave.tgz -- slave log/coredump file

=== KEYDB BUG REPORT START: Cut & paste starting from here ===
8:S 20 May 2019 05:33:31.930 # === ASSERTION FAILED ===
8:S 20 May 2019 05:33:31.930 # ==> server.cpp:4927 'false' is not true
8:S 20 May 2019 05:33:31.930 # (forcing SIGSEGV to print the bug report.)
8:S 20 May 2019 05:33:31.930 # KeyDB 0.9.5 crashed by signal: 11
8:S 20 May 2019 05:33:31.930 # Crashed running the instruction at: 0x555db4f32cae
8:S 20 May 2019 05:33:31.930 # Accessing address: 0xffffffffffffffff
8:S 20 May 2019 05:33:31.930 # Failed assertion: false (server.cpp:4927)

------ STACK TRACE ------
EIP:
/usr/local/bin/keydb-server *:9086(_serverAssert+0x7e) [0x555db4f32cae]

Backtrace:
/usr/local/bin/keydb-server :9086(logStackTrace(ucontext_t)+0x6b) [0x555db4f35c9b]
/usr/local/bin/keydb-server :9086(sigsegvHandler(int, siginfo_t, void*)+0xbf) [0x555db4f3642f]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x12890) [0x7fbf3525a890]
/usr/local/bin/keydb-server *:9086(_serverAssert+0x7e) [0x555db4f32cae]
/usr/local/bin/keydb-server *:9086(OnTerminate()+0x1c) [0x555db4ee2bdc]
/usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0x92ab6) [0x7fbf35aafab6]
/usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0x92af1) [0x7fbf35aafaf1]
/usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0x92d24) [0x7fbf35aafd24]
/usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0x8ea19) [0x7fbf35aaba19]
/usr/local/bin/keydb-server :9086(writeToClient(int, client, int)+0x4e5) [0x555db4efa345]
/usr/local/bin/keydb-server *:9086(handleClientsWithPendingWrites(int)+0x183) [0x555db4efa723]
/usr/local/bin/keydb-server :9086(beforeSleepLite(aeEventLoop)+0x101) [0x555db4ee3101]
/usr/local/bin/keydb-server *:9086(aeMain+0x75) [0x555db4edee45]
/usr/local/bin/keydb-server :9086(workerThreadMain(void)+0x98) [0x555db4ee35a8]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7fbf3524f6db]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x7fbf34f7888f]

------ INFO OUTPUT ------
8:S 20 May 2019 05:33:31.932 # === ASSERTION FAILED ===
8:S 20 May 2019 05:33:31.932 # ==> object.cpp:961 'GlobalLocksAcquired()' is not true
8:S 20 May 2019 05:33:31.932 # (forcing SIGSEGV to print the bug report.)
9:C 20 May 2019 05:33:32.449 * Before turning into a replica, using my master parameters to synthesize a cached master: I may be able to synchronize with the new master with just a partial transfer.
9:C 20 May 2019 05:33:32.449 * Notice: "active-replica yes" implies "replica-read-only no"
9:C 20 May 2019 05:33:32.450 # oO0OoO0OoO0Oo KeyDB is starting oO0OoO0OoO0Oo
9:C 20 May 2019 05:33:32.450 # KeyDB version=0.9.5, bits=64, commit=4333615b, modified=0, pid=9, just started
9:C 20 May 2019 05:33:32.450 # Configuration loaded

[JohnSully]

Awesome! That pinpointed the problem. I should have a fix soon.

[JohnSully]

I was never able to reproduce the actual issue, but I was able to find the bug from code review. The issue is we try to unlock the client's mutex twice. std::unique_lock throws in this situation.

Fixed in change: 7617a87

[hellower]

I recreated docker container using "eqalpha/keydb:unstable" docker image.

It usually crashed in less than 5 minutes after starting, but it is now stable.
Thank you for your quick fix.
It's stable version for me.

[JohnSully]

Glad its resolved! This will be in the next stable release.