Ubuntu 24_04 testSCCMLTests1_openj9_0_FAILED Cannot remove shared cache "ShareClassesCMLTests" as there are JVMs still attached to the cache about openj9

[pshipton]

Failed 100% on ubu24x86-beta-svl-rt1-1, passed on other machines (ubu22x86-svl-rt3-1, ubu22x86-svl-rt8-1, rhel7x86-rtp-rt2-1, cent8x86-rtp-rt3-1).
@hangshao0 pls take a look.

[hangshao0]

[2024-03-20T00:55:01.093Z] [ERR] JVMSHRC012I Cannot remove shared cache "ShareClassesCMLTests" as there are JVMs still attached to the cache

This means number of attached process to the shared memory is not 0.

Tries a simple command 3 times, seems that cacheInfo->nattach value is garbage value (8717, 9215, 9696).

jenkins@ubu24x86-beta-svl-rt2-1:~/workspace$ java -Xtrace:print={j9shr.2317,j9prt.1052,j9prt.56-60} -Xshareclasses:nonpersistent,name=Cache0 -version
16:17:56.412*0x17000           j9prt.56       > j9shmem_stat entered, name = C290M11F1A64_memory_Cache0_G43L00
16:17:56.412 0x17000           j9prt.60       < j9shmem_stat exit - return success
16:17:56.413 0x17000           j9prt.56       > j9shmem_stat entered, name = C290M11F1A64_memory_Cache0_G43L00
16:17:56.413 0x17000           j9prt.60       < j9shmem_stat exit - return success
16:17:56.415 0x17000           j9shr.2317     < SH_OSCachesysv::getCacheStatsHelper: Exit. cacheInfo->name is Cache0, cacheInfo->os_shmid is 655627, cacheInfo->lastattach is 1711037876000, cacheInfo->lastdetach is 1711037874000, cacheInfo->nattach is 8717
...
jenkins@ubu24x86-beta-svl-rt2-1:~/workspace$ java -Xtrace:print={j9shr.2317,j9prt.1052,j9prt.56-60} -Xshareclasses:nonpersistent,name=Cache0 -version
16:17:58.419*0x17000           j9prt.56       > j9shmem_stat entered, name = C290M11F1A64_memory_Cache0_G43L00
16:17:58.419 0x17000           j9prt.60       < j9shmem_stat exit - return success
16:17:58.419 0x17000           j9prt.56       > j9shmem_stat entered, name = C290M11F1A64_memory_Cache0_G43L00
16:17:58.419 0x17000           j9prt.60       < j9shmem_stat exit - return success
16:17:58.421 0x17000           j9shr.2317     < SH_OSCachesysv::getCacheStatsHelper: Exit. cacheInfo->name is Cache0, cacheInfo->os_shmid is 655627, cacheInfo->lastattach is 1711037878000, cacheInfo->lastdetach is 1711037876000, cacheInfo->nattach is 9215
....
jenkins@ubu24x86-beta-svl-rt2-1:~/workspace$ java -Xtrace:print={j9shr.2317,j9prt.1052,j9prt.56-60} -Xshareclasses:nonpersistent,name=Cache0 -version
16:18:58.938*0x17000           j9prt.56       > j9shmem_stat entered, name = C290M11F1A64_memory_Cache0_G43L00
16:18:58.938 0x17000           j9prt.60       < j9shmem_stat exit - return success
16:18:58.939 0x17000           j9prt.56       > j9shmem_stat entered, name = C290M11F1A64_memory_Cache0_G43L00
16:18:58.939 0x17000           j9prt.60       < j9shmem_stat exit - return success
16:18:58.941 0x17000           j9shr.2317     < SH_OSCachesysv::getCacheStatsHelper: Exit. cacheInfo->name is Cache0, cacheInfo->os_shmid is 655627, cacheInfo->lastattach is 1711037938000, cacheInfo->lastdetach is 1711037878000, cacheInfo->nattach is 9696

While, on another machine, cacheInfo->nattach is consistent.

[pshipton]

We need to figure out what's wrong with Ubuntu 24_04, or this particular machine. It's still in beta, but we're getting close to the release date.

• are configuration changes required?
• are code changes required to support Ubuntu 24_04?
• is it no longer possible to support Ubuntu 24_04 for non-persistent caches?

I didn't check what build we have installed, but here I see a daily build.

[hangshao0]

Using ipcs, I see the nattch of the shared memory is 12116, even when there is no java process attached to it.

key        shmid      owner      perms      bytes      nattch     status
0x6100771b 655627     jenkins    600        314572800  12116

Then I run a simple command:
java -Xtrace:print={j9shr.2317,j9prt.56,j9prt.57-60} -Xshareclasses:nonpersistent,name=Cache0 -version
After it finished, ipcs shows the nattch becomes 12742

key        shmid      owner      perms      bytes      nattch     status
0x6100771b 655627     jenkins    600        314572800  12742

I tried using ipcrm -m to remove the shared memory, but it cannot be removed. I guess it is because nattch > 0. It is marked as dest, which means it will be deleted when all processes attached to it terminated.

------ Shared Memory Segments --------
key        shmid      owner      perms      bytes      nattch     status
0x00000000 655627     jenkins    600        314572800  12742      dest

It looks to me the nattch reported by the OS is incorrect. It is hard for me to believe there are 12742 processes attached to this shared memory.

There are over 200 shared memory segments that are marked 'dest', but not deleted on this machine.

[pshipton]

Added a note to infrastructure/issues/8836