Server repeatedly crashes during "reading device" after restart about aerospike-server

[fillest]

I just thought - notice how "loaded .. object .." line cuts on ... /dev/disk/by-id/? - is it possible the problem is in some logging buffer overflow on the stack and not disk data corruption? I have quite a long list of disk partitions here and use full serial-number-paths so the whole line gets close-ish to a len of 1000, is there maybe some char[1000] buffer somewhere? Can I somehow check this in gdb for the coredump? (I'm very bad at it)

If I replace disk paths with shorter symbolic names (e.g. /dev/nvme1n1p1) with a different order - will Aerospike understand the reordering? (OS assigns this names randomly on each boot)

[kportertx]

Yes, we do see a problem with the drive ticker in that version - it seems to have been inadvertently resolved in 5.x when the format of that line was changed.

How many devices are configured for this namespace?

The buffer that is being filled in that ticker is 2048 bytes - though logging will truncate that to 1024 bytes. Looking at your device names and assuming your serial numbers are 32 bytes (which is how long mine appear to be), seems that you would need at least 34 devices configured to overflow this buffer. If you do have more that 34 devices configured then I'd recommend upgrading to 5.x+ to quickly mitigate this issue.

Note that 4.9 is mentioned as a pre-req to upgrade to 5.x. Server 4.9 included 5.x compatibility changes for XDR which is EE only but there was also changes to how we replicate records which also required that we go through a certain set of compatible versions. Server 4.6, had the replication compatibility code in place already, so on Community you should be able to upgrade from 4.6 directly to 5.x.

Also, there were some configuration changes related to eviction and expiration mentioned in the 4.9.0.3 release notes which may require that you modify your /etc/aerospike/aerospike.conf.

[fillest]

32 devices, yes. My serial numbers are 33 bytes.
We cannot afford to upgrade to a major version yet at this time. Thanks for comment on that though.
I guess the safest/easiest way now will be to re-partition or to try symlinks with shorter names. I guess after I test this I should close the ticket as the problem is fixed in 5.х