munmap_chunk(): invalid pointer, "nuraft_w_0" received signal SIGABRT, Aborted about nuraft

[greensky00]

Hi @kishorekrd

Looks like you are using BoringSSL. Do you see the same issue even with OpenSSL?

[kishorekrd]

Yes, I have seen 3 more different crashes in BoringSSL while running NuRaft. I did not change any NuRaft configuration. I built NuRaft with default configuration. I don't see BoringSSL installed on my system. Here is the list of ssl packages

$ sudo dpkg-query -l | grep -i ssl
ii  libcurl4:amd64                         7.58.0-2ubuntu3.12                          amd64        easy-to-use client-side URL transfer library (OpenSSL flavour)
ii  libcurl4-openssl-dev:amd64             7.58.0-2ubuntu3.12                          amd64        development files and documentation for libcurl (OpenSSL flavour)
ii  libflac8:amd64                         1.3.2-1                                     amd64        Free Lossless Audio Codec - runtime C library
ii  libio-socket-ssl-perl                  2.060-3~ubuntu18.04.1                       all          Perl module implementing object oriented interface to SSL sockets
ii  libnet-smtp-ssl-perl                   1.04-1                                      all          Perl module providing SSL support to Net::SMTP
ii  libnet-ssleay-perl                     1.84-1ubuntu0.2                             amd64        Perl module for Secure Sockets Layer (SSL)
ii  libssl-dev:amd64                       1.1.1-1ubuntu2.1~18.04.9                    amd64        Secure Sockets Layer toolkit - development files
ii  libssl1.0.0:amd64                      1.0.2n-1ubuntu5.6                           amd64        Secure Sockets Layer toolkit - shared libraries
ii  libssl1.1:amd64                        1.1.1-1ubuntu2.1~18.04.9                    amd64        Secure Sockets Layer toolkit - shared libraries
ii  libwavpack1:amd64                      5.1.0-2ubuntu1.5                            amd64        audio codec (lossy and lossless) - library
ii  libxmlsec1-openssl:amd64               1.2.25-1build1                              amd64        Openssl engine for the XML security library
ii  libzstd1:amd64                         1.3.3+dfsg-2ubuntu1.2                       amd64        fast lossless compression algorithm
**ii  openssl                                1.1.1-1ubuntu2.1~18.04.9                    amd64        Secure Sockets Layer toolkit - cryptographic utility**
ii  perl-openssl-defaults:amd64            3build1                                     amd64        version compatibility baseline for Perl OpenSSL packages
ii  python3-certifi                        2018.1.18-2                                 all          root certificates for validating SSL certs and verifying TLS hosts (python3)
ii  python3-openssl                        17.5.0-1ubuntu1                             all          Python 3 wrapper around the OpenSSL library
ii  python3-service-identity               16.0.0-2                                    all          Service identity verification for pyOpenSSL (Python 3 module)

[greensky00]

I think it is compiled with a custom library, not from the system path.

OPENSSL_free (orig_ptr=0x7fff480018f8) at external/boringssl/src/crypto/mem.c:154

What is the CMake message (when you do cmake ..) for SSL library path? It should be like this if it uses the system library:

-- Open SSL library path: /usr/lib/x86_64-linux-gnu/libssl.a

If it is not, you may manually add the path /usr/lib/x86_64-linux-gnu to CMakeLists.txt, so as to make it scanned first:

    find_path(OPENSSL_LIBRARY_PATH
              NAMES libssl.a
              PATHS /usr/lib/x86_64-linux-gnu
                    ${PROJECT_SOURCE_DIR}
                    ${DEPS_PREFIX}/lib
                    ${DEPS_PREFIX}/lib64
                    /usr/local/opt/openssl/lib
                    ${LIB_PATH_HINT})

[kishorekrd]

Yes, I see it

-- deps prefix is not given
-- Open SSL library path: /usr/lib/x86_64-linux-gnu/libssl.a
-- Output library file name: libnuraft.a
-- Configuring done
-- Generating done

Even though NuRaft is built with openssl, it is somehow using boringssl. Is there anyway I can force Nuraft to use openssl?

[greensky00]

From what you shared, I think the below is the case

• Your NuRaft is built with OpenSSL (it is a static library so that NuRaft itself doesn't include OpenSSL objects).
• But, your executable binary (using NuRaft) is linked with BoringSSL.

To avoid such wrong linkage, you may need to

• Build NuRaft with BoringSSL, or
• Use OpenSSL for your executable binary.

[kishorekrd]

Thanks for the response. If I don't want to use SSL right now, Can I use mock_ssl by disabling SSL ?
cmake -DDISABLE_SSL=1 ../

[greensky00]

Yes you can. But does this happen even though you don't set below enable_ssl_ option?

asio_service::options asio_opt;
asio_opt.enable_ssl_ = true;

https://github.com/eBay/NuRaft/blob/master/docs/enabling_ssl.md

[kishorekrd]

Yes that's correct. I have this following settings

asio_service::options asio_opt;
asio_opt.enable_ssl_ = false;

What is the default behavior?

[greensky00]

Default is false. I just wonder if this issue happens with the false option.

[kishorekrd]

Even with false, the issue is happening. I think the main issue is mixing openssl and Boringssl.
As you mentioned
To avoid such wrong linkage, you may need to

Build NuRaft with BoringSSL, or
Use OpenSSL for your executable binary.

[kishorekrd]

Hi, Even though I disabled SSL for Nuraft build, I am still seeing asio running some ssl code.

Jun 13 20:39:27 -- Build type is not given, use default.
Jun 13 20:39:27 -- Build type: RelWithDebInfo
Jun 13 20:39:27 -- Build Install Prefix : /home/user/thirdparty/NuRaft/install
Jun 13 20:39:27 -- ASIO include path: /home/azureuser/work2/barrel/thirdparty/NuRaft/asio/asio/include
Jun 13 20:39:27 -- deps prefix is not given
Jun 13 20:39:27 -- ---- DISABLED SSL ----
Jun 13 20:39:27 -- Output library file name: libnuraft.a

How to disable SSL completely for Nuraft and also asio?

How can I build Nuraft with Boringssl?
I see flag OPENSSL_IS_BORINGSSL in asio code. How to use this flag?

[greensky00]

@kishorekrd
If -DDISABLE_SSL option is on, NuRaft does not even include SSL library (LIBSSL must be empty):

NuRaft/CMakeLists.txt

Lines 130 to 139 in 714db11

	if (NOT OPENSSL_LIBRARY_PATH)
	message(STATUS "Use system's side OpenSSL library")
	set(LIBSSL ssl)
	set(LIBCRYPTO crypto)
	else ()
	message(STATUS "Open SSL library path: ${OPENSSL_LIBRARY_PATH}/libssl.a")
	set(LIBSSL ${OPENSSL_LIBRARY_PATH}/libssl.a)
	set(LIBCRYPTO ${OPENSSL_LIBRARY_PATH}/libcrypto.a)
	endif ()
	endif ()

You can check symbols in libnuraft.a:

$ nm libnuraft.a -C | grep ssl

If it is compiled correctly, the only thing you can see should be mock_ssl.