RocksDB.loadLibrary() cause project crash about rocksdb

[ajkr]

It is difficult to guess. How did you build /tmp/librocksdbjni3872642940403144838.so?

[zhangstar333]

It is difficult to guess. How did you build /tmp/librocksdbjni3872642940403144838.so?

get it from rocksdbjni-8.9.1.jar.

[ajkr]

Thanks. How was /install_data/ldb_toolchain built?

[adamretter]

I am afraid that I have no idea from the stack trace. All I can say is that RocksDB.loadLibrary() does very little apart from ask the JVM to load the native C++ library, it does not yet open the database or make any calls to RocksDB.

I am not sure what you mean by:

my project is java + cpp, so use a reflect to invoke the function,

Can you elaborate please?

[zhangstar333]

Thanks. How was /install_data/ldb_toolchain built?

download from this project: https://github.com/amosbird/ldb_toolchain_gen

[zhangstar333]

I am afraid that I have no idea from the stack trace. All I can say is that RocksDB.loadLibrary() does very little apart from ask the JVM to load the native C++ library, it does not yet open the database or make any calls to RocksDB.

I am not sure what you mean by:

my project is java + cpp, so use a reflect to invoke the function,

Can you elaborate please?

enenen. like this:

a java function,
public void java_function() {
........
RocksDB.loadLibrary();
......
}

and in c++ function, will call above java function like this:

void c++_function() {
jclass jclazz = env->GetObjectClass(thiz);
jmethodID jmethodID = env->GetMethodID(jclazz, "java_function", "()V");
env->CallVoidMethod(thiz, jmethodID);
}
and I have test, if remove the line RocksDB.loadLibrary();, it could work well.
But if loadLibrary, it's will be carsh, I don't know if it's related to use jemalloc in my c++ project?

[adamretter]

I don't know if it's related to use jemalloc in my c++ project?

Have you tried without JEMalloc?

[zhangstar333]

I don't know if it's related to use jemalloc in my c++ project?

Have you tried without JEMalloc?

so bad, I test again without jemalloc, it's still crash by same error of allocation-size-too-big

[adamretter]

Are you able to build a debug version of the RocksDB library using the flag PORTABLE=0 and then translate the stacktrace offsets to line numbers in the source code (see https://github.com/facebook/rocksdb/wiki/JNI-Debugging#stack).

I think if you could provide the line and location of the following two offsets from the stacktrace, that can be useful:

#6 0x561b74003e8e in std::map<rocksdb::LevelStatType, rocksdb::LevelStat, std::less<rocksdb::LevelStatType>, std::allocator<std::pair<rocksdb::LevelStatType const, rocksdb::LevelStat> > >::map(std::initializer_list<std::pair<rocksdb::LevelStatType const, rocksdb::LevelStat> >, std::less<rocksdb::LevelStatType> const&, std::allocator<std::pair<rocksdb::LevelStatType const, rocksdb::LevelStat> > const&) 
#7 0x7f7447de8492  (/tmp/librocksdbjni3872642940403144838.so+0x1ef492)

[zhangstar333]

@adamretter I have a demo about this case, and you could test it. I record the reproduction steps,

1. build java code by: javac -cp "rocksdbjni-8.9.1.jar" ForkTest.java
2. build C++ code by: clang++ ForkTest.cpp -I/mnt/disk2/zhangsida/install_data/jdk1.8.0_131/include -I/mnt/disk2/zhangsida/install_data/jdk1.8.0_131/include/linux -ldl -lpthread -std=c++17 -o forktest
3. run by: ./forktest java

[VM-10-8-centos]$ ./forktest java
path: /mnt/disk2/zhangsida/install_data/jdk1.8.0_131
JVM created.
c++ main begin.
start to load rocksdb
Java object created error.

ForkTest.cpp

#include <jni.h>
#include <unistd.h>
#include <stdlib.h>
#include <stdio.h>
#include <iostream>
#include <string>
#include <memory>
#include <mutex>
#include <filesystem>
#include <utility>
#include <dlfcn.h>

class LibJVMLoader {
public:
    LibJVMLoader(const LibJVMLoader&) = delete;
    LibJVMLoader& operator=(const LibJVMLoader&) = delete;

    static LibJVMLoader& instance();
    int load();

    using JNI_GetCreatedJavaVMsPointer = std::add_pointer_t<decltype(::JNI_GetCreatedJavaVMs)>;
    static JNI_GetCreatedJavaVMsPointer JNI_GetCreatedJavaVMs;

    using JNI_CreateJavaVMPointer = std::add_pointer_t<decltype(::JNI_CreateJavaVM)>;
    static JNI_CreateJavaVMPointer JNI_CreateJavaVM;

private:
    explicit LibJVMLoader(std::string library)
            : _library(std::move(library)), _handle(nullptr, nullptr) {}

    const std::string _library;
    std::unique_ptr<void, void (*)(void*)> _handle;
};


_JNI_IMPORT_OR_EXPORT_ jint JNICALL JNI_GetCreatedJavaVMs(JavaVM** vm_buf, jsize bufLen,
                                                          jsize* numVMs) {
    return LibJVMLoader::JNI_GetCreatedJavaVMs(vm_buf, bufLen, numVMs);
}

_JNI_IMPORT_OR_EXPORT_ jint JNICALL JNI_CreateJavaVM(JavaVM** pvm, void** penv, void* args) {
    return LibJVMLoader::JNI_CreateJavaVM(pvm, penv, args);
}

namespace {

#ifndef __APPLE__
#define LIBJVM_SO "libjvm.so"
#else
#define LIBJVM_SO "libjvm.dylib"
#endif

template <typename T>
int resolve_symbol(T& pointer, void* handle, const char* symbol) {
    pointer = reinterpret_cast<T>(dlsym(handle, symbol));
    return (pointer != nullptr)
                   ? 0
                   : -1;
}

} // namespace


LibJVMLoader::JNI_GetCreatedJavaVMsPointer LibJVMLoader::JNI_GetCreatedJavaVMs = nullptr;
LibJVMLoader::JNI_CreateJavaVMPointer LibJVMLoader::JNI_CreateJavaVM = nullptr;

LibJVMLoader& LibJVMLoader::instance() {
    static std::once_flag find_library;
    static std::string library;
    std::call_once(find_library, []() {
        const auto* java_home = getenv("JAVA_HOME");
        if (!java_home) {
            return;
        }
        std::string path(java_home);
        for (const auto& entry : std::filesystem::recursive_directory_iterator(path)) {
            if (entry.path().filename() == LIBJVM_SO) {
                library = entry.path().string();
                break;
            }
        }
    	fprintf(stderr, "path: %s\n", path.c_str());
    });

    static LibJVMLoader loader(library);
    return loader;
}

int LibJVMLoader::load() {
    if (_library.empty()) {
    	fprintf(stderr, "_library is empty\n");
        return -1;
    }

    static std::once_flag resolve_symbols;
    static int status;
    std::call_once(resolve_symbols, [this]() {
        _handle = std::unique_ptr<void, void (*)(void*)>(dlopen(_library.c_str(), RTLD_LAZY),
                                                         [](void* handle) { dlclose(handle); });
        if (!_handle) {
            status = -1;
            return;
        }

        if (-1 == resolve_symbol(JNI_GetCreatedJavaVMs, _handle.get(), "JNI_GetCreatedJavaVMs")) {
            status = -1;
            return;
        }
        if (-1 == resolve_symbol(JNI_CreateJavaVM, _handle.get(), "JNI_CreateJavaVM")) {
            status = -1;
            return;
        }
    });
    return status;
}

// Create the Java virtual machine, and start a test object.
static JNIEnv* createJVM(void) {
    if (LibJVMLoader::instance().load() < 0) {
    	return nullptr;
    }
    JavaVM* jvm;
    JNIEnv* env;

    JavaVMOption options[20];
    int numOptions = 0;
    options[numOptions++].optionString = "-Djava.class.path=.";
    options[numOptions++].optionString = "-Xss1m";

    JavaVMInitArgs args;
    args.version = JNI_VERSION_1_2;
    args.options = options;
    args.nOptions = (jint)numOptions;
    args.ignoreUnrecognized = 0;

    if (JNI_CreateJavaVM(&jvm, (void **)&env, &args)) {
    	fprintf(stderr, "Can't create JVM.\n");
    }
    fprintf(stderr, "JVM created.\n");
    return env;
}

void testJava(JNIEnv *env) {
    jclass clazz = env->FindClass("ForkTest");
    if (env->ExceptionOccurred() || (clazz == 0)) {
    	fprintf(stderr, "Can't find Java class.\n");
        exit(1); 
    }
    jmethodID init = env->GetMethodID(clazz, "<init>", "()V");
    if (env->ExceptionOccurred() || (init == 0)) {
    	fprintf(stderr, "Can't find Java method.\n");
        exit(1);
    }
    jobject obj = env->NewObject(clazz, init);
    if (env->ExceptionOccurred() || (obj == 0)) {
    	fprintf(stderr, "Java object created error.\n");
        exit(1);
    }
}

int main(int argc, char **argv) {
    JNIEnv* env = createJVM();
    if (!env) {
    	exit(1);
    }
    fprintf(stderr, "c++ main begin.\n");
    if (argc > 1) {
        testJava(env);
    }
    fprintf(stderr, "c++ main end.\n");
    return 0;
}

ForkTest.java

import java.io.*;
import org.rocksdb.RocksDB;

public class ForkTest {
    public static void main(String[] args) throws Exception {
	    new ForkTest();
    }

    public ForkTest() {
	    exec();
    }

    public void exec() {
        System.out.println("start to load rocksdb");
        RocksDB.loadLibrary();
        System.out.println("end to load rocksdb");
    }
}

uname -a
Linux VM-10-8-centos 5.4.119-19-0009.3 #1 SMP Wed Apr 20 22:27:43 CST 2022 x86_64 x86_64 x86_64 GNU/Linux

clang++ --version
clang version 15.0.7 (Red Hat 15.0.7-1.module+el8.8.0+452+cdf25910)
Target: x86_64-koji-linux-gnu
Thread model: posix
InstalledDir: /usr/bin

 ./java -version
java version "1.8.0_131"
Java(TM) SE Runtime Environment (build 1.8.0_131-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.131-b11, mixed mode)

[ajkr]

This doesn't seem to repro what the original issue is about, plus I tried changing the classpath to options[numOptions++].optionString = "-Djava.class.path=.:rocksdbjni-8.9.1-linux64.jar"; and it worked

$ ./forktest java
path: /usr/lib/jvm/java-8-openjdk-amd64
JVM created.
c++ main begin.
start to load rocksdb
end to load rocksdb
c++ main end.

$ uname -a
Linux ip-172-31-3-137 6.2.0-1017-aws #17~22.04.1-Ubuntu SMP Fri Nov 17 21:07:13 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
$ java -version
openjdk version "1.8.0_392"
OpenJDK Runtime Environment (build 1.8.0_392-8u392-ga-1~22.04-b08)
OpenJDK 64-Bit Server VM (build 25.392-b08, mixed mode)
$ clang++ --version
Ubuntu clang version 17.0.5 (++20231114093119+98bfdac5ce82-1~exp1~20231114093217.68)
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /home/ubuntu/install/ldb_toolchain/bin

[zhangstar333]

very thanks! I will have a try.