Assertion failed in testJITServer_0: Pointer for offset could not be found in deserializer shared cache about openj9

[cjjdespres]

Attn @mpirvu.

[cjjdespres]

I think I found the problem. Just to summarize the cause - I was working under the assumption that if the JVM ever unloads a class loader, then all of the classes it loaded would have been unloaded already at that point. That does not seem to be true.

Here is roughly what happened in the log:

#JITServer: Associated class loader 00007F103409B388 with class jit/test/tr/chtable/VirtualGuardTest$A chain 0000000000000000

#JITServer: Cached class loader record ID 4 -> { 00007F103409B388 } for first loaded class jit/test/tr/chtable/VirtualGuardTest$A
#JITServer: Cached class record ID 325 -> { 00000000004FA200, 4 } for class jit/test/jitserver/JITServerTest$1
#JITServer: Cached class record ID 326 -> { 00000000004CD400, 1 } for class java/io/FilenameFilter
#JITServer: Cached class chain record ID 249 -> { 00000000004FA200 } for class jit/test/jitserver/JITServerTest$1 ID 325
#JITServer: Deserialized AOT method jit/test/jitserver/JITServerTest$1.<init>(Ljava/lang/String;)V
#JITServer: JITClient: Applying remote AOT relocations to deserialized newly compiled AOT body for jit/test/jitserver/JITServerTest$1.<init>(Ljava/lang/String;)V @ warm

#JITServer: Removed class loader 00007F103409B388 associated with class jit/test/tr/chtable/VirtualGuardTest$A chain 0000000000000000
#JITServer: Invalidated class loader 00007F103409B388 ID 4 in the deserializer cache

#JITServer: Associated class loader 00007F1035D35768 with class jit/test/tr/chtable/VirtualGuardTest$A chain 0000000000000000
#JITServer: Client sending compReq seqNo=19685 to server for method jit/test/tr/chtable/VirtualGuardTest$A.<init>(Ljit/test/tr/chtable/VirtualGuardTest;)V @ warm.
#JITServer: ERROR: Failed to deserialize AOT method jit/test/tr/chtable/VirtualGuardTest$A.<init>(Ljit/test/tr/chtable/VirtualGuardTest;)V
! (AOT warm) jit/test/tr/chtable/VirtualGuardTest$A.<init>(Ljit/test/tr/chtable/VirtualGuardTest;)V Q_SZ=0 Q_SZI=0 QW=6 j9m=00000000006AC228 time=2943us aotCacheDeserializationFailure memLimit=262144 KB compThreadID=0

#JITServer: Client sending compReq seqNo=19729 to server for method jit/test/jitserver/JITServerTest$1.<init>(Ljava/lang/String;)V @ warm.
#JITServer: Deserialized AOT method jit/test/jitserver/JITServerTest$1.<init>(Ljava/lang/String;)V
#JITServer: JITClient: Applying remote AOT relocations to deserialized AOT body for jit/test/jitserver/JITServerTest$1.<init>(Ljava/lang/String;)V @ warm

<crash occurs around here>

So:

1. The client associates a class loader to a name in the class loader table
2. The server compiles an AOT warm method that references it, and the client successfully deserializes it
3. The client removes/invalidates that class loader in the class loader table and deserializer cache
4. The client associates a new class loader to that same name
5. The client starts compiling methods linked to that class loader. Usually they fail to be deserialized - I assume this is because they fail in class record deserialization, because we marked the class loader as invalid in (3). (There should probably be a verbose log line here indicating that fact - I'll add that).
6. The client requests that a method with the same name as (2) be compiled. Looking at the client log, it seems like none of the classes referenced in the deserialization of (2) were unloaded/invalidated, so I'm assuming that class/class chain record deserialization succeeded without ever examining the cached class loader entry (which should have been marked as invalid in (3)).
7. During relocation, we try to find the referenced class loader and hit the assert, because the cached class loader record is marked invalid (the class loader pointer is NULL).

I think the solution is simply to introduce a check here:

openj9/runtime/compiler/runtime/JITServerAOTDeserializer.cpp

Lines 1057 to 1060 in 0db560f

	if (it->second._ramClass)
	{
	return true;
	}

that also makes sure that the associated cached class loader record is still valid.

[cjjdespres]

I've tried out that fix, and it seems to avoid the crash. Now, however, I get a timeout in that test - the client isn't destroyed within 30s at the end of testServerUnreachableForAWhile and testServerComesUpAfterClient. That seems unrelated, so I'll take a look at that separately.

[cjjdespres]

Actually, I think that might be the pre-existing #19084.

[cjjdespres]

I got other crashes during testing that made me realize that the fix in #19461 (comment) and a subsequent initial attempt in the PR #19472 were incorrect.

The crashes I was getting were similar to the above, but caused by previously-invalidated class records in the deserializer instead. These were happening during ROM class or class chain queries in TR_J9DeserializerSharedCache.

The invalidated records were not caught in the deserializer because we only call cacheRecord() on records that are actually sent over by the server. Since the server keeps track of records it knows the client has, it will not send over every single record that a method needs. I knew this, but failed to realize that meant that I couldn't have cached record revalidation take place in the various cacheRecord() methods. Instead, this revalidation needs to occur in updateSCCOffsets, where we can ensure that any cached serialization record needed during subsequent relocation is still valid.