Possible memory leak about hermes

[neildhar]

It is possible that the issue you're observing is a known leak in WeakMap, which was discussed in #1147 and fixed in e7b2abe.

Could you try calling gc() from JS periodically and see if that brings the memory consumption under control?

[lucaswitch]

After running the application for 13 hours this crash is ocurring.

[lucaswitch]

Also can see this on the stack trace, not sure how to reproduce it yet cause i can't find the actual cause of the crash on the js code.

[tmikov]

The error in the previous screenshot seems to be out of memory?

[lucaswitch]

@tmikov yes, also the cpu goes all the way up when the crash begins.

[tmikov]

Everybody is on a break now, but we will take a more serious look after the holidays.

[lucaswitch]

@tmikov thank you for replying and happy holidays!

[lucaswitch]

Any updates on this stacktrace?
@tmikov

[neildhar]

Hey @lucaswitch, we'd need more information, like a minimal repro to offer more insight here. It looks like you may be running into two separate problems, the first is a crash somewhere in the GC, and the second is an OOM.

The OOM is likely caused by a large number of allocations, and you can try debugging it by connecting to the application in DevTools and taking a heap snapshot or an allocation timeline profile. That should help you pin down where the allocations are coming from. It is also worth monitoring the runtime's reporting memory consumption by calling HermesInternal.getInstrumentedStats() in JS.

The crash is trickier and may indicate a bug in Hermes, unless Hermes is being used incorrectly (e.g. from multiple threads simultaneously). Are you able to share a minimised JS repro that results in the crash?

[lucaswitch]

Thank you for replying, I'm also debugging for possible causes that could create the OOM or any kind of over memory/cpu usage.
I will try to create a reproducer once i figure this out. The problem is i can't figure out where it is and sometimes it needs a couple hours of use to reproduce the crash.

[lucaswitch]

@neildhar Does this getInstrumentedStats looks normal:

{"js_VMExperiments": 0, "js_allocatedBytes": 26423128, "js_gcCPUTime": 0.123, "js_gcTime": 0.4050000000000003, "js_heapSize": 33554432, "js_mallocSizeEstimate": 0, "js_markStackOverflows": 0, "js_numGCs": 224, "js_totalAllocatedBytes": 903991048, "js_vaSize": 33554432}

js_totalAllocatedBytes looks too big but not sure if it's to be that way on debug mode.

Is there a documentation about the getInstrumentedStats attributes meaning?

[tmikov]

32MB heap, looks totally normal.

[lucaswitch]

So i think the oom is solved(we did a full refactor of the app), but the crash sadly persists the same.

Also now i'm keeping eye on the getInstrumentedStats all the time and seems normal(only js_totalAllocatedBytes gets quite big)

I will consider that some module could be messing up the runtime for now.
Is there some kind of object that could provide which native modules are using jsi at the moment of a crash?

I'm looking some variables at the moment of the crash to indicate which module could be causing this crash then i can take a deeper look at It. And then make a repro(or remove it).

Looking after those stacktrace i could not find enough information that could lead a third party module.

The stacktrace changes a little bit between crashes except on LINE facebook::jsi::RuntimeDecorator::call (decorator.h:340) and above.
I can always see the variables and debug into it, and thats what ive doing.

What we know at this moment:
The methodId and moduleId at the moment of the crash are always between these values:

moduleId: JSTimers and methodId callTimers

moduleId RCTEventEmitter and methodId ReceiveTouches

moduleId RCTEventEmitter and methodId emit

Can you provide some help to find It on how to find which module it's being executed?

[lucaswitch]

The issue also only happens on iOS 16 and 17. Our app works with user health monitoring so we had to shutdown the iOS version to solve that issue due to thousands of this crash.

[tmikov]

@lucaswitch

Is there a documentation about the getInstrumentedStats attributes meaning?

@neildhar can probably provide more detail, but most of the fields are documented here:

hermes/include/hermes/VM/GCBase.h

Line 365 in ff6e5a9

struct HeapInfo {

The other fields just show the total number of GCs and how much time in milliseconds they took.

[lucaswitch]

@lucaswitch

Is there a documentation about the getInstrumentedStats attributes meaning?

@neildhar can probably provide more detail, but most of the fields are documented here:

hermes/include/hermes/VM/GCBase.h

Line 365 in ff6e5a9

struct HeapInfo {

The other fields just show the total number of GCs and how much time in milliseconds they took.

Thank you this header file contains all that i need about the getInstrumentedStats.

Can you provide some help which files contains the information about which react-native native module is calling the jsi environment?

[lucaswitch]

Found something! This code from watermelon database heavily access the jsi runtime.
But i don't think there is any issue with that since it does not open another thread.
Here is the code:

#include "Database.h"
#include "DatabasePlatform.h"
#include "JSIHelpers.h"

namespace watermelondb {

using platform::consoleError;
using platform::consoleLog;

void Database::install(jsi::Runtime *runtime) {
    jsi::Runtime &rt = *runtime;
    auto globalObject = rt.global();
    createMethod(rt, globalObject, "nativeWatermelonCreateAdapter", 2, [runtime](jsi::Runtime &rt, const jsi::Value *args) {
        std::string dbPath = args[0].getString(rt).utf8(rt);
        bool usesExclusiveLocking = args[1].getBool();

        jsi::Object adapter(rt);

        std::shared_ptr<Database> database = std::make_shared<Database>(runtime, dbPath, usesExclusiveLocking);
        adapter.setProperty(rt, "database", jsi::Object::createFromHostObject(rt, database));

        // FIXME: Important hack!
        // Without any hacks, JSI Watermelon crashes on Android/Hermes on app reload in development:
        // (This doesn't happen on iOS/JSC)
        //   abort 0x00007d0bd27cff2f
        //   __fortify_fatal(char const*, ...) 0x00007d0bd27d20c1
        //   HandleUsingDestroyedMutex(pthread_mutex_t*, char const*) 0x00007d0bd283b020
        //   pthread_mutex_lock 0x00007d0bd283aef4
        //   pthreadMutexEnter sqlite3.c:26320
        //   sqlite3_mutex_enter sqlite3.c:25775
        //   sqlite3_next_stmt sqlite3.c:84221
        //   watermelondb::SqliteDb::~SqliteDb() Sqlite.cpp:57
        // It appears that the Unix thread on which Database is set up is already destroyed by the
        // time destructor is called. AFAIU destructors on objects that are managed by JSI runtime
        // *should* be safe in this respect, but maybe they're not/there's a bug...
        //
        // For future debuggers, the flow goes like this:
        //  - ReactInstanceManager.runCreateReactContextOnNewThread()
        //       this sets up new instance
        //  - ReactInstanceManager.tearDownReactContext()
        //  - ReactContext.destroy()
        //  - CatalystInstanceImpl.destroy()
        //       this notifies listeners that the app is about to be destroyed
        //  - mHybridData.resetNative()
        //  - ~CatalystInstanceImpl()
        //  - ~Instance()
        //  - NativeToJSBridge.destroy()
        //  - m_executor = nullptr
        //  - ~Runtime()
        //  - ...
        //  - ~Database()
        //
        // First attempt to work around this issue was by disabling sqlite3's threadsafety (which caused
        // pthread apis to be called, leading to a crash), since we're only using it from one thread
        // but predictably that caused new issues.
        // When using headless JS, this issue would occur:
        //    Failed to get a row for query - sqlite error 11 (database disk image is malformed)
        // (Not exactly sure why, seems like headless JS reuses the same catalyst instance...)
        //
        // Current workaround is to tap into CatalystInstanceImpl.destroy() to destroy the database
        // before it's destructed via normal C++ rules. There's no clean API for our JSI setup, so
        // we route via NativeModuleRegistry onCatalystInstanceDestroy -> DatabaseBridge ->
        // WatermelonJSI via reflection (and switch to the currect thread - important!) and then to
        // individual Database objects via this listener callback. It's ugly, but should work.
        //
        // 2023 update: Check if the above is still true, given https://github.com/Nozbe/WatermelonDB/issues/1474
        // showed that the true cause of the pthread_mutex_lock crash is something else.
        // On the other hand, it's still true that invalidation happens asynchronously and could happen
        // after new bridge is already set up, which could cause locking issues (and a case was found on iOS where
        // this does happen)
        std::weak_ptr<Database> weakDatabase = database;
        platform::onDestroy([weakDatabase]() {
            if (auto databaseToDestroy = weakDatabase.lock()) {
                consoleLog("Destroying database due to RCTBridge invalidation");
                databaseToDestroy->destroy();
            }
        });

        createMethod(rt, adapter, "initialize", 2, [database](jsi::Runtime &rt, const jsi::Value *args) {
            jsi::String dbName = args[0].getString(rt);
            int expectedVersion = (int)args[1].getNumber();

            int databaseVersion = database->getUserVersion();

            jsi::Object response(rt);

            if (databaseVersion == expectedVersion) {
                database->initialized_ = true;
                response.setProperty(rt, "code", "ok");
            } else if (databaseVersion == 0) {
                response.setProperty(rt, "code", "schema_needed");
            } else if (databaseVersion < expectedVersion) {
                response.setProperty(rt, "code", "migrations_needed");
                response.setProperty(rt, "databaseVersion", databaseVersion);
            } else {
                consoleLog("Database has newer version (" + std::to_string(databaseVersion) +
                           ") than what the app supports (" + std::to_string(expectedVersion) + "). Will reset database.");
                response.setProperty(rt, "code", "schema_needed");
            }

            return response;
        });
        createMethod(rt, adapter, "setUpWithSchema", 3, [database](jsi::Runtime &rt, const jsi::Value *args) {
            jsi::String dbName = args[0].getString(rt);
            jsi::String schema = args[1].getString(rt);
            int schemaVersion = (int)args[2].getNumber();

            try {
                database->unsafeResetDatabase(schema, schemaVersion);
            } catch (const std::exception &ex) {
                consoleError("Failed to set up the database correctly - " + std::string(ex.what()));
                std::abort();
            }

            database->initialized_ = true;
            return jsi::Value::undefined();
        });
        createMethod(rt, adapter, "setUpWithMigrations", 4, [database](jsi::Runtime &rt, const jsi::Value *args) {
            jsi::String dbName = args[0].getString(rt);
            jsi::String migrationSchema = args[1].getString(rt);
            int fromVersion = (int)args[2].getNumber();
            int toVersion = (int)args[3].getNumber();

            try {
                database->migrate(migrationSchema, fromVersion, toVersion);
            } catch (const std::exception &ex) {
                consoleError("Failed to migrate the database correctly - " + std::string(ex.what()));
                return makeError(rt, ex.what());
            }

            database->initialized_ = true;
            return jsi::Value::undefined();
        });
        createMethod(rt, adapter, "find", 2, [database](jsi::Runtime &rt, const jsi::Value *args) {
            assert(database->initialized_);
            jsi::String tableName = args[0].getString(rt);
            jsi::String id = args[1].getString(rt);
            return database->find(tableName, id);
        });
        createMethod(rt, adapter, "query", 3, [database](jsi::Runtime &rt, const jsi::Value *args) {
            assert(database->initialized_);
            jsi::String tableName = args[0].getString(rt);
            jsi::String sql = args[1].getString(rt);
            jsi::Array arguments = args[2].getObject(rt).getArray(rt);
            return database->query(tableName, sql, arguments);
        });
        createMethod(rt, adapter, "queryAsArray", 3, [database](jsi::Runtime &rt, const jsi::Value *args) {
            assert(database->initialized_);
            jsi::String tableName = args[0].getString(rt);
            jsi::String sql = args[1].getString(rt);
            jsi::Array arguments = args[2].getObject(rt).getArray(rt);
            return database->queryAsArray(tableName, sql, arguments);
        });
        createMethod(rt, adapter, "queryIds", 2, [database](jsi::Runtime &rt, const jsi::Value *args) {
            assert(database->initialized_);
            jsi::String sql = args[0].getString(rt);
            jsi::Array arguments = args[1].getObject(rt).getArray(rt);
            return database->queryIds(sql, arguments);
        });
        createMethod(rt, adapter, "unsafeQueryRaw", 2, [database](jsi::Runtime &rt, const jsi::Value *args) {
            assert(database->initialized_);
            jsi::String sql = args[0].getString(rt);
            jsi::Array arguments = args[1].getObject(rt).getArray(rt);
            return database->unsafeQueryRaw(sql, arguments);
        });
        createMethod(rt, adapter, "count", 2, [database](jsi::Runtime &rt, const jsi::Value *args) {
            assert(database->initialized_);
            jsi::String sql = args[0].getString(rt);
            jsi::Array arguments = args[1].getObject(rt).getArray(rt);
            return database->count(sql, arguments);
        });
        createMethod(rt, adapter, "batch", 1, [database](jsi::Runtime &rt, const jsi::Value *args) {
            assert(database->initialized_);
            jsi::Array operations = args[0].getObject(rt).getArray(rt);
            database->batch(operations);
            return jsi::Value::undefined();
        });
        createMethod(rt, adapter, "batchJSON", 1, [database](jsi::Runtime &rt, const jsi::Value *args) {
            assert(database->initialized_);
            database->batchJSON(args[0].getString(rt));
            return jsi::Value::undefined();
        });
        createMethod(rt, adapter, "getLocal", 1, [database](jsi::Runtime &rt, const jsi::Value *args) {
            assert(database->initialized_);
            jsi::String key = args[0].getString(rt);
            return database->getLocal(key);
        });
        createMethod(rt, adapter, "unsafeLoadFromSync", 4, [database](jsi::Runtime &rt, const jsi::Value *args) {
            assert(database->initialized_);
            auto jsonId = (int) args[0].getNumber();
            auto schema = args[1].getObject(rt);
            auto preamble = args[2].getString(rt).utf8(rt);
            auto postamble = args[3].getString(rt).utf8(rt);
            return database->unsafeLoadFromSync(jsonId, schema, preamble, postamble);
        });
        createMethod(rt, adapter, "unsafeExecuteMultiple", 1, [database](jsi::Runtime &rt, const jsi::Value *args) {
            assert(database->initialized_);
            auto sqlString = args[0].getString(rt).utf8(rt);
            database->executeMultiple(sqlString);
            return jsi::Value::undefined();
        });
        createMethod(rt, adapter, "unsafeResetDatabase", 2, [database](jsi::Runtime &rt, const jsi::Value *args) {
            assert(database->initialized_);
            jsi::String schema = args[0].getString(rt);
            int schemaVersion = (int)args[1].getNumber();

            try {
                database->unsafeResetDatabase(schema, schemaVersion);
                return jsi::Value::undefined();
            } catch (const std::exception &ex) {
                consoleError("Failed to reset database correctly - " + std::string(ex.what()));
                // Partially reset database is likely corrupted, so it's probably less bad to crash
                std::abort();
            }
        });
        createMethod(rt, adapter, "unsafeClose", 0, [database](jsi::Runtime &rt, const jsi::Value *args) {
            assert(database->initialized_);
            database->destroy();
            database->initialized_ = false;
            return jsi::Value::undefined();
        });

        return adapter;
    });

    // TODO: Use the onMemoryAlert hook!
}


} // namespace watermelondb

[neildhar]

which react-native native module is calling the jsi environment

That would fall outside Hermes unfortunately. It is possible that RN has some logging that logs transitions into native code from JS, but we can't provide much input there.

Your best bet may be to attach a debugger and set breakpoints on entry into the Hermes runtime to determine if anything is potentially corrupting runtime state.

[lucaswitch]

Can you provide some interesting points on React-jsi directory to place the debugger on?
Having a hard time to where to place it

[tmikov]

This is probably a good place, you should get a hit whenever native calls into JavaScript:

hermes/API/hermes/hermes.cpp

Line 2160 in ff6e5a9

jsi::Value HermesRuntimeImpl::call(

[lucaswitch]

Is it normal to have this quantity of leak warnings on debug mode?

[neildhar]

@lucaswitch According to Apple's documentation it looks like the leak detector works by scanning the application memory for pointers to regions that were allocated with malloc.

The behaviour you're seeing is expected, since we recently made a change to how the pointers are encoded in the heap, which would result in the leaks tool not being able to find them. However, this isn't a real leak, and Hermes should still release the memory whenever the ArrayBuffer is garbage collected.

[lucaswitch]

Thank you for the fast response! Ok i will keep looking around on jsi modules.

[lucaswitch]

The stacktrace changed a bit. Removed a lot of the application deps.

[lucaswitch]

Done! @neildhar. As the #1147 i placed it every 100ms.

[lucaswitch]

Another crash, also i can see that the Module being called is the JSTimers.

[lucaswitch]

Also i figure out that this is the current third party modules accessing the jsi functions.

[tmikov]

@lucaswitch the top of the stack looks a deadlock exception thrown by the sampling profiler mutex. Very interesting. Can we see the message of that exception?

[lucaswitch]

Here is another one maybe related to #1203.
Also because our app uses redux-saga it relies heavily on generator functions. I can see on the stacktrace some breadcrumbs that could be that: hermes::vm::GeneratorInnerFunction

Also i was not able to get the exception pointer value...

[lucaswitch]

Solved! Closing as this issue is not a hermes engine issue.
It was a private react-native module with c++ responsible for doing fast fourrier transform that was causing the issue(maybe some wrong offset access or any kind o dynamic resource allocation).
After a full rewrite of this module in plain javascript the crash was gone.

Thank you for all the support on this issue. @tmikov @neildhar
Very much appreciated.