HeapHogLoadTest_5m_0_FAILED Segmentation error vmState=0x00000000 about openj9

[gacholio]

That contradicts your statement:

Due to the J2I frame, JIT has setup the arguments for [invokebasic](https://github.com/eclipse-openj9/openj9/blob/master/runtime/vm/BytecodeInterpreter.hpp#L9246) and the fromJIT code-path needs to be taken in [invokebasic](https://github.com/eclipse-openj9/openj9/blob/master/runtime/vm/BytecodeInterpreter.hpp#L9246). Taking the interpreter path has probably led to the crash.

as we are running in an interpreted method that was invoked by the JIT.

[nbhuiyan]

@babsingh

Due to the J2I frame, JIT has setup the arguments for invokebasic

The JIT would not have set up the arguments for invokebasic. The tempSlot value that would have been set by the compiled code for invokebasic calls would be indicating the number of stack slots occupied by the args, and therefore would be a positive number.

[nbhuiyan]

are the arguments on the stack being populated from the JIT?

Given that the previous frame is a jitted method, yes.

[pshipton]

@knn-k can you pls take the first look at the core.

[pshipton]

@babsingh fyi in case it's related to #14713

[knn-k]

It is a SEGV caused by an instruction trying to load from an address pointed by x10, which holds 0x99669966 eyecatcher value.

   8630c: 08 17 40 f9  	ldr	x8, [x24, #40]
   86310: 14 01 40 f9  	ldr	x20, [x8]
   86314: ea 03 14 aa  	mov	x10, x20
   86318: 4b 0d 9f b8  	ldrsw	x11, [x10, #-16]! <- SEGV here, x10=x20=0x99669966
   8631c: 4a 01 0b 8b  	add	x10, x10, x11
   86320: 0b 03 40 f9  	ldr	x11, [x24]
   86324: 6b 15 4c f9  	ldr	x11, [x11, #6184]
   86328: 4c 01 40 79  	ldrh	w12, [x10]
   8632c: 8d 69 6a 38  	ldrb	w13, [x12, x10]
   86330: bf 6d 01 71  	cmp	w13, #91
   86334: a0 01 00 54  	b.eq	0x86368 <__ZN26VM_BytecodeInterpreterFull3runEP10J9VMThread+0x14fe0>

[knn-k]

x20 is loaded from [x8], and jdmpview shows x8 is a pointer to a J9Class MethodTypeForm as shown below.

> !j9class 0x14169b100
J9Class at 0x14169b100 {
  Fields for J9Class:
	0x0: UDATA eyecatcher = 0x0000000099669966 (2573637990)
	0x8: struct J9ROMClass* romClass = !j9romclass 0x0000000121429CA8
	0x10: void** superclasses = !j9x 0x000000014169AFE0
	0x18: UDATA classDepthAndFlags = 0x00000000020E0001 (34471937)
	0x20: U32 classDepthWithFlags = 0x00000000 (0)
	0x24: U32 classFlags = 0x00000000 (0)
	0x28: struct J9ClassLoader* classLoader = !j9classloader 0x000000013C088668
	0x30: struct J9Object* classObject = !j9object 0x00000002800087A0 // java/lang/Class
	0x38: volatile UDATA initializeStatus = 0x0000000000000001 (1)
	0x40: struct J9Method* ramMethods = !j9method 0x000000014169B318 // java/lang/invoke/MethodTypeForm.erasedType()Ljava/lang/invoke/MethodType;
	0x48: UDATA* ramStatics = !j9x 0x000000014169B518
	0x50: struct J9Class* arrayClass = !j9class 0x0000000000000000
	0x58: UDATA totalInstanceSize = 0x0000000000000030 (48)
	0x60: struct J9ITable* lastITable = !j9itable 0x0000000100BC7878
	0x68: UDATA* instanceDescription = !j9x 0x000000000000003D
	0x70: UDATA* instanceLeafDescription = !j9x 0x0000000000000001
	0x78: UDATA instanceHotFieldDescription = 0x0000000000000004 (4)
	0x80: UDATA selfReferencingField1 = 0x0000000000000000 (0)
	0x88: UDATA selfReferencingField2 = 0x0000000000000000 (0)
	0x90: struct J9Method* initializerCache = !j9method 0x0000000000000000
	0x98: UDATA romableAotITable = 0x000000010209C000 (4329160704)
	0xa0: UDATA packageID = 0x0000000121391F01 (4852358913)
	0xa8: struct J9Module* module = !j9module 0x000000013C089868
	0xb0: struct J9Class* subclassTraversalLink = !j9class 0x000000014169AA00 // java/lang/invoke/LambdaForm
	0xb8: struct J9Class* subclassTraversalReverseLink = !j9class 0x000000014169C100 // jdk/internal/vm/Continuation
	0xc0: void** iTable = !j9x 0x0000000000000000
	0xc8: UDATA castClassCache = 0x00000001416BC701 (5392549633)
	0xd0: void** jniIDs = !j9x 0x0000000000000000
	0xd8: UDATA lockOffset = 0x0000000000000008 (8)
	0xe0: U32 paddingForGLRCounters = 0x00000000 (0)
	0xe4: U16 reservedCounter = 0x0000 (0)
	0xe6: U16 cancelCounter = 0x0000 (0)
	0xe8: UDATA newInstanceCount = 0x00000000000003E8 (1000)
	0xf0: IDATA backfillOffset = 0x0000000000000038 (56)
	0xf8: struct J9Class* replacedClass = !j9class 0x0000000000000000
	0x100: UDATA finalizeLinkOffset = 0x0000000000000000 (0)
	0x108: struct J9Class* nextClassInSegment = !j9class 0x000000014169AA00 // java/lang/invoke/LambdaForm
	0x110: UDATA* ramConstantPool = !j9x 0x000000014169ACA0
	0x118: struct J9Object** callSites = !j9x 0x0000000000000000
	0x120: struct J9Object** invokeCache = !j9x 0x0000000000000000
	0x128: struct J9Object** varHandleMethodTypes = !j9x 0x0000000000000000
	0x130: struct J9VMCustomSpinOptions* customSpinOption = !j9vmcustomspinoptions 0x0000000000000000
	0x138: void** staticSplitMethodTable = !j9x 0x0000000000000000
	0x140: void** specialSplitMethodTable = !j9x 0x0000000000000000
	0x148: struct J9JITExceptionTable* jitMetaDataList = !j9jitexceptiontable 0x0000000000000000
	0x150: struct J9Class* gcLink = !j9class 0x0000000000000000
	0x158: struct J9Class* hostClass = !j9class 0x000000014169B100 // java/lang/invoke/MethodTypeForm
	0x160: struct J9Class* nestHost = !j9class 0x0000000000000000
	0x168: struct J9FlattenedClassCache* flattenedClassCache = !j9flattenedclasscache 0x0000000000000000
	0x170: struct J9ClassHotFieldsInfo* hotFieldsInfo = !j9classhotfieldsinfo 0x0000000000000000
	0x178: struct J9MemberNameListNode* memberNames = !j9membernamelistnode 0x0000000000000000
}
Class name: java/lang/invoke/MethodTypeForm
To view static fields, use !j9statics 0x000000014169B100
To view instance shape, use !j9classshape 0x000000014169B100

[knn-k]

Many registers in the register dump have the addresses of a specific region.
0x1705C27A0-0x1705C27D8 in x1, x4, x5, x6, x11, x12, x13, x21, x23, x24, x25, x26, x27, x29

The memory dump looks like a table of addresses.

• The x8 value for MethodTypeForm above was loaded from 0x1705c27d8.
• 0x13c019e20 at 0x1705c27b0 is j9javavm.
• 0x110d3ed1c at 0x1705c27c8 is a jitReturnAddress in crashInfo. It is in the address range of JITed invokeVirtual().

> hexdump 0x1705C27A0 80

1705c27a0: 30285c70 01000000 7c93ae00 01000000  |0(\p....|.......|
1705c27b0: 209e013c 01000000 0045963c 01000000  | ..<.....E.<....|
1705c27c0: a0000a34 01000000 1cedd310 01000000  |...4............|
1705c27d0: 00000000 00000000 00b16941 01000000  |..........iA....|
1705c27e0: 01000000 00000000 10000000 00000000  |................|

> !whatis 0x013c019e20
Found 0x000000013C019E20 as !j9javavm 0x13c019e20
Match found
> !whatis 0x0110d3ed1c
Found 0x0000000110D3ED1C as !void 0x110d3ed1c: !j9javavm 0x13c019e20->j9ras->crashInfo->failingThread->linkNext->jitReturnAddress
Match found

> info jitm
...
        start=0x110d3ec68  end=0x110d3edd0   java/lang/invoke/LambdaForm$DMH/0x000000003c99c820::invokeVirtual(Ljava/lang/Object;Ljava/lang/Object;I)F
...

[babsingh]

@knn-k DDR cmds to find the location of the crash in the interpreter:

> !gpinfo
Failing Thread: !j9vmthread 0x13c95ef00
Failing Thread ID: 0x14797b8a (343505802)
....

> !stackslots 0x13c95ef00
<13c95ef00> *** BEGIN STACK WALK, flags = 00400001 walkThread = 5,311,426,304 ***
<13c95ef00> 	ITERATE_O_SLOTS
<13c95ef00> 	RECORD_BYTECODE_PC_OFFSET
<13c95ef00> Initial values: walkSP = 0x0000000134810CB0, PC = 0x000000013622C451, literals = 0x0000000136819588, A0 = 0x0000000134810D88, j2iFrame = 0x0000000134810D78, decomp = 0x0000000000000000
<13c95ef00> J2I frame: bp = 0x0000000134810D78, sp = 0x0000000134810CB0, pc = 0x000000013622C451, cp = 0x00000001368194C0, arg0EA = 0x0000000134810D88, flags = 0x0000000010000000
<13c95ef00> 	Method: java/lang/invoke/LambdaForm$NFI/0x000000003622c320.invoke_LI_F(Ljava/lang/invoke/MethodHandle;[Ljava/lang/Object;)Ljava/lang/Object; !j9method 0x0000000136819588
<13c95ef00> 	Bytecode index = 13 <--- Bytecode being executed
<13c95ef00> 	Using local mapper
<13c95ef00> 	Locals starting at 0x0000000134810D88 for 0x0000000000000002 slots
<13c95ef00> 		I-Slot: a0[0x0000000134810D88] = 0x00000002805D0080
<13c95ef00> 		I-Slot: a1[0x0000000134810D80] = 0x000000036C3169F8
May 08, 2024 6:23:59 PM com.ibm.j9ddr.vm29.events.DefaultEventListener corruptData
WARNING: CorruptData encountered iterating o-slots. walkThread = 0x000000013C95EF00
com.ibm.j9ddr.CorruptDataException: Operand stack underflow in StackMap
	at com.ibm.j9ddr.vm29.j9.stackmap.StackMap$J9MappingStack.POP(StackMap.java:184)
	at com.ibm.j9ddr.vm29.j9.stackmap.StackMap$StackMap_V1.outputStackMap(StackMap.java:728)
	at com.ibm.j9ddr.vm29.j9.stackmap.StackMap$StackMap_V1.j9stackmap_StackBitsForPC(StackMap.java:271)
	at com.ibm.j9ddr.vm29.j9.stackmap.StackMap.j9stackmap_StackBitsForPC(StackMap.java:106)
	....

<13c95ef00> JIT frame: bp = 0x0000000134810DC8, pc = 0x0000000110521538, unwindSP = 0x0000000134810D80, cp = 0x000000013C138AC0, arg0EA = 0x0000000134810DE0, jitInfo = 0x0000000141A9A9E8
<13c95ef00> 	Method: java/lang/invoke/LambdaForm$DMH/0x000000003c09ac20.invokeStatic(Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object; !j9method 0x000000013C138BA8
<13c95ef00> 	Bytecode index = 11, inlineDepth = 0, PC offset = 0x0000000000000110
<13c95ef00> 	stackMap=0x0000000141A9AB04, slots=I16(0x0003) parmBaseOffset=I16(0x0008), parmSlots=U16(0x0003), localBaseOffset=I16(0xFFD8)
<13c95ef00> 	Described JIT args starting at 0x0000000134810DD0 for U16(0x0003) slots
<13c95ef00> 		O-Slot: : a2[0x0000000134810DD0] = 0x000000036C3169F8
<13c95ef00> 		O-Slot: : a1[0x0000000134810DD8] = 0x00000002805D0080
<13c95ef00> 		O-Slot: : a0[0x0000000134810DE0] = 0x0000000280BEAD58
<13c95ef00> 	Described JIT temps starting at 0x0000000134810DA0 for IDATA(0x0000000000000005) slots
<13c95ef00> 		O-Slot: : t4[0x0000000134810DA0] = 0x0000000000000000
...

<13c95ef00> JIT inline frame: bp = 0x0000000134810E68, pc = 0x00000001109CA128, unwindSP = 0x0000000134810DD0, cp = 0x00000001501302C0, arg0EA = 0x0000000000000000, jitInfo = 0x000000016007D7E8
<13c95ef00> 	Method: java/lang/invoke/LambdaForm$NamedFunction.invokeWithArguments([Ljava/lang/Object;)Ljava/lang/Object; !j9method 0x0000000150130A10
<13c95ef00> 	Bytecode index = 21, inlineDepth = 1, PC offset = 0x00000001109C9DA7
<13c95ef00> JIT frame: bp = 0x0000000134810E68, pc = 0x00000001109CA128, unwindSP = 0x0000000134810DD0, cp = 0x0000000141698B00, arg0EA = 0x0000000134810E80, jitInfo = 0x000000016007D7E8
<13c95ef00> 	Method: java/lang/invoke/LambdaForm.interpretName(Ljava/lang/invoke/LambdaForm$Name;[Ljava/lang/Object;)Ljava/lang/Object; !j9method 0x000000014169A3F8
<13c95ef00> 	Bytecode index = 123, inlineDepth = 0, PC offset = 0x0000000000000338
<13c95ef00> 	stackMap=0x000000016007DAE4, slots=I16(0x0003) parmBaseOffset=I16(0x0008), parmSlots=U16(0x0003), localBaseOffset=I16(0xFFC8)
<13c95ef00> 	Described JIT args starting at 0x0000000134810E70 for U16(0x0003) slots
<13c95ef00> 		O-Slot: : a2[0x0000000134810E70] = 0x000000036C316998
<13c95ef00> 		O-Slot: : a1[0x0000000134810E78] = 0x0000000280BECFD0
<13c95ef00> 		O-Slot: : a0[0x0000000134810E80] = 0x0000000280BECEC8
<13c95ef00> 	Described JIT temps starting at 0x0000000134810E30 for IDATA(0x0000000000000007) slots
<13c95ef00> 		I-Slot: : t6[0x0000000134810E30] = 0x0000000000000001
...

> !j9method 0x0000000136819588
J9Method at 0x136819588 {
  Fields for J9Method:
	0x0: U8* bytecodes = !j9x 0x000000013622C444 // "*+2+2�"
	0x8: struct J9ConstantPool* constantPool = !j9constantpool 0x00000001368194C0 (flags = 0x0)
	0x10: void* methodRunAddress = !j9x 0x0000000000000006
	0x18: volatile void* extra = !j9x 0x0000000000000F93
}
Signature: java/lang/invoke/LambdaForm$NFI/0x000000003622c320.invoke_LI_F(Ljava/lang/invoke/MethodHandle;[Ljava/lang/Object;)Ljava/lang/Object; !bytecodes 0x0000000136819588
ROM Method: !j9rommethod 0x000000013622C430
Next Method: !j9method 0x00000001368195A8

> !bytecodes 0x0000000136819588
  Name: invoke_LI_F
  Signature: (Ljava/lang/invoke/MethodHandle;[Ljava/lang/Object;)Ljava/lang/Object;
  Access Flags (20080008): default static
  Internal Attribute Flags: @FrameIteratorSkip
  ...

    0 aload0
    ...
    7 checkcast 3 java/lang/Integer
   10 invokevirtual 4 java/lang/Integer.intValue()I
   13 invokevirtual 6 java/lang/invoke/MethodHandle.invokeBasic(Ljava/lang/Object;I)F <--- Bytecode being run
   16 invokestatic 8 java/lang/Float.valueOf(F)Ljava/lang/Float;
   19 return1

> !j9vmthread 0x13c95ef00
J9VMThread at 0x13c95ef00 {
  Fields for J9VMThread:
        ...
	0x20: UDATA* sp = !j9x 0x0000000134810CB0
	0x28: U8* pc = !j9x 0x000000013622C451 // "�"
	0x30: struct J9Method* literals = !j9method 0x0000000136819588 // java/lang/invoke/LambdaForm$NFI/0x000000003622c320.invoke_LI_F(Ljava/lang/invoke/MethodHandle;[Ljava/lang/Object;)Ljava/lang/Object;
	0x38: UDATA jitStackFrameFlags = 0x0000000000000000 (0)
	0x40: struct J9Object* jitException = !j9object 0x000000036BFDA050 // java/lang/ArrayIndexOutOfBoundsException
	0x48: struct J9Object* currentException = !j9object 0x0000000000000000
	0x50: UDATA* stackOverflowMark = !j9x 0xFFFFFFFFFFFFFFFF
	0x58: UDATA* stackOverflowMark2 = !j9x 0x000000013480CC28
        ...
	0xe8: UDATA tempSlot = 0xFFFFFFFFFFFFFFA8 (-88)
	0xf0: void* jitReturnAddress = !j9x 0x0000000110988410
	0xf8: void* floatTemp1 = !j9x 0x0000000000000000
	0x100: void* floatTemp2 = !j9x 0x00000000000CFB1C
	0x108: void* floatTemp3 = !j9x 0x0000000110C01C68
	0x110: void* floatTemp4 = !j9x 0x0000000000000000
	0x118: UDATA returnValue = 0x000000000000000D (13)
	0x120: UDATA returnValue2 = 0x0000000280618598 (10743809432)
        ...

[babsingh]

in case it's related to #14713

I don't see the symptoms from #14713 (comment). J9VMThread->literals is good in this crash. There are similarities in terms of tempSlot = -88 and invokebasic being run.

But, this crash looks like the reverse of #14713 (comment).

• In this case, there is a J2I frame at the top, and we are running the invokebasic INL.
• While running the invokebasic INL, J9VMThread->jitStackFrameFlags is 0. The fromJIT code-path won't be taken in invokebasic.
• Due to the J2I frame, JIT has setup the arguments for invokebasic and the fromJIT code-path needs to be taken in invokebasic. Taking the interpreter path has probably led to the crash.
• Also, DDR indicates corrupt data while processing the J2I frame. So, there is probably a bad O-slot in the J2I frame.

fyi @nbhuiyan @jdmpapin @0xdaryl @gacholio

[gacholio]

If there is a J2I frame, then the jitStackFrameFlags have already been consumed. Is the invokebasic coming from the interpreted method that's running in the J2I frame?

[babsingh]

Is the invokebasic coming from the interpreted method that's running in the J2I frame?

Yes, bytecode 13 in the J2I frame is an invokebasic.

[babsingh]

I see. So, we are taking the !fromJIT code-path in invokeBasic, which is correct since an interpreted method is being run.

For the interpreted method (invoke_LI_F) in the J2I frame, are the arguments on the stack being populated from the JIT?

<13c95ef00> J2I frame: bp = 0x0000000134810D78, sp = 0x0000000134810CB0, pc = 0x000000013622C451, cp = 0x00000001368194C0, arg0EA = 0x0000000134810D88, flags = 0x0000000010000000
<13c95ef00> 	Method: java/lang/invoke/LambdaForm$NFI/0x000000003622c320.invoke_LI_F(Ljava/lang/invoke/MethodHandle;[Ljava/lang/Object;)Ljava/lang/Object; !j9method 0x0000000136819588
<13c95ef00> 	Bytecode index = 13
<13c95ef00> 	Using local mapper
<13c95ef00> 	Locals starting at 0x0000000134810D88 for 0x0000000000000002 slots
<13c95ef00> 		I-Slot: a0[0x0000000134810D88] = 0x00000002805D0080
<13c95ef00> 		I-Slot: a1[0x0000000134810D80] = 0x000000036C3169F8

[babsingh]

The crash happened on an amac: https://na.artifactory.swg-devops.com/artifactory/sys-rt-generic-local/hyc-runtimes-jenkins.swg-devops.com/Test_openjdk22_j9_extended.system_aarch64_mac_testList_0/10/system_test_output.tar.gz. I am unable to view the native stack in lldb. Can anyone see the line of code in lldb where the crash happened?