crui fails to work on some machines about openj9

[pshipton]

@TobiAjila fyi

[tajila]

@ymanton Do you know if the playbooks were run externally as well?

[ymanton]

As of this morning, yes (aside from cent8-aarch64-* machines which can't build CRIU for some reason - I'm looking into it, and ub16x64j91 which is down).

Based on the above I think something went wrong. It looks like a newer CRIU was installed (because it is complaining about being run as non-root without cap-checkpoint-restore; old versions weren't caps-aware) but the caps weren't set properly on the binary. I'll take a look at this as well.

[pshipton]

Related?

https://openj9-jenkins.osuosl.org/job/Test_openjdk21_j9_sanity.functional_x86-64_linux_OpenJDK21_testList_0/13 - cent7-x64-3

00:32:29  Testing: Envvar test7
00:32:29  Test start time: 2024/02/22 00:32:27 Eastern Standard Time
00:32:29  Running command: bash /home/jenkins/workspace/Test_openjdk21_j9_sanity.functional_x86-64_linux_OpenJDK21_testList_0/aqa-tests/TKG/../../jvmtest/functional/cmdLineTests/criu/criuScript.sh /home/jenkins/workspace/Test_openjdk21_j9_sanity.functional_x86-64_linux_OpenJDK21_testList_0/aqa-tests/TKG/../../jvmtest/functional/cmdLineTests/criu /home/jenkins/workspace/Test_openjdk21_j9_sanity.functional_x86-64_linux_OpenJDK21_testList_0/jdkbinary/j2sdk-image/bin/java " -XX:+DebugOnRestore -Xjit:count=0  -Xtrace:print=j9vm.735" org.openj9.criu.EnvVarFileTest EnvVarFileTest7 1
00:32:29  Time spent starting: 4 milliseconds
00:32:48  Time spent executing: 18758 milliseconds
00:32:48  Test result: FAILED
00:32:48  Output from test:
00:32:48   [OUT] start running script
00:32:48   [OUT] export GLIBC_TUNABLES=glibc.cpu.hwcaps=-XSAVEC,-XSAVE,-AVX2,-ERMS,-AVX,-AVX_Fast_Unaligned_Load
00:32:48   [OUT] export LD_BIND_NOT=on
00:32:48   [OUT] /home/jenkins/workspace/Test_openjdk21_j9_sanity.functional_x86-64_linux_OpenJDK21_testList_0/jdkbinary/j2sdk-image/bin/java -XX:+EnableCRIUSupport  -XX:+DebugOnRestore -Xjit:count=0  -Xtrace:print=j9vm.735 -cp /home/jenkins/workspace/Test_openjdk21_j9_sanity.functional_x86-64_linux_OpenJDK21_testList_0/aqa-tests/TKG/../../jvmtest/functional/cmdLineTests/criu/criu.jar org.openj9.criu.EnvVarFileTest EnvVarFileTest7 1
00:32:48   [OUT] Pre-checkpoint
00:32:48   [OUT] Performing CRIUSupport.checkpointJVM(), current thread name: main, Thu Feb 22 00:32:38 EST 2024, System.currentTimeMillis(): 1708579963707, System.nanoTime(): 25521009731800788
00:32:48   [OUT] JVMJITM044W Some or all compiled code in the code cache invalidated post restore.
00:32:48   [OUT] Post-checkpoint
00:32:48   [OUT] Removed test output files
00:32:48   [OUT] finished script
00:32:48  >> Success condition was not found: [Output match: Killed]
00:32:48  >> Failure condition was not found: [Output match: failed properties test]
00:32:48  >> Required condition was found: [Output match: Pre-checkpoint]
00:32:48  >> Success condition was not found: [Output match: Failed to load options from the options file]
00:32:48  >> Failure condition was not found: [Output match: CRIU is not enabled]
00:32:48  >> Failure condition was not found: [Output match: Operation not permitted]
00:32:48  >> Success condition was not found: [Output match: Thread pid mismatch]
00:32:48  >> Success condition was not found: [Output match: do not match expected]
00:32:48  >> Success condition was not found: [Output match: Unable to create a thread:]
00:32:48  >> Failure condition was not found: [Output match: Could not dump the JVM processes, err=-70]
00:32:48  >> Failure condition was not found: [Output match: Restore arg: -Dprop1=val1]
00:32:48  >> Failure condition was not found: [Output match: Restore arg: -Dprop2=val2]
00:32:48  >> Failure condition was not found: [Output match: Restore arg: -Dprop3=val3]
00:32:48  >> Failure condition was not found: [Output match: User requested Java dump using]

00:33:24  Testing: Envvar test10
00:33:24  Test start time: 2024/02/22 00:33:23 Eastern Standard Time
00:33:24  Running command: bash /home/jenkins/workspace/Test_openjdk21_j9_sanity.functional_x86-64_linux_OpenJDK21_testList_0/aqa-tests/TKG/../../jvmtest/functional/cmdLineTests/criu/criuScript.sh /home/jenkins/workspace/Test_openjdk21_j9_sanity.functional_x86-64_linux_OpenJDK21_testList_0/aqa-tests/TKG/../../jvmtest/functional/cmdLineTests/criu /home/jenkins/workspace/Test_openjdk21_j9_sanity.functional_x86-64_linux_OpenJDK21_testList_0/jdkbinary/j2sdk-image/bin/java " -XX:+DebugOnRestore -Xjit:count=0 " org.openj9.criu.EnvVarFileTest EnvVarFileTest10 1
00:33:24  Time spent starting: 5 milliseconds
00:33:44  Time spent executing: 18486 milliseconds
00:33:44  Test result: FAILED
00:33:44  Output from test:
00:33:44   [OUT] start running script
00:33:44   [OUT] export GLIBC_TUNABLES=glibc.cpu.hwcaps=-XSAVEC,-XSAVE,-AVX2,-ERMS,-AVX,-AVX_Fast_Unaligned_Load
00:33:44   [OUT] export LD_BIND_NOT=on
00:33:44   [OUT] /home/jenkins/workspace/Test_openjdk21_j9_sanity.functional_x86-64_linux_OpenJDK21_testList_0/jdkbinary/j2sdk-image/bin/java -XX:+EnableCRIUSupport  -XX:+DebugOnRestore -Xjit:count=0  -cp /home/jenkins/workspace/Test_openjdk21_j9_sanity.functional_x86-64_linux_OpenJDK21_testList_0/aqa-tests/TKG/../../jvmtest/functional/cmdLineTests/criu/criu.jar org.openj9.criu.EnvVarFileTest EnvVarFileTest10 1
00:33:44   [OUT] Pre-checkpoint
00:33:44   [OUT] Performing CRIUSupport.checkpointJVM(), current thread name: main, Thu Feb 22 00:33:34 EST 2024, System.currentTimeMillis(): 1708580019572, System.nanoTime(): 25521065595969023
00:33:44   [OUT] JVMJITM044W Some or all compiled code in the code cache invalidated post restore.
00:33:44   [OUT] JVMJ9VM007E Command-line option unrecognised: -Xmx64m
00:33:44   [OUT] Exception in thread "main" org.eclipse.openj9.criu.JVMRestoreException: The JVM could not enable all the restore options specified
00:33:44   [OUT] 	at openj9.criu/org.eclipse.openj9.criu.CRIUSupport.checkpointJVM(CRIUSupport.java:517)
00:33:44   [OUT] 	at org.openj9.criu.CRIUTestUtils.checkPointJVM(CRIUTestUtils.java:77)
00:33:44   [OUT] 	at org.openj9.criu.EnvVarFileTest.envVarFileTest10(EnvVarFileTest.java:290)
00:33:44   [OUT] 	at org.openj9.criu.EnvVarFileTest.main(EnvVarFileTest.java:66)
00:33:44   [OUT] Caused by: openj9.internal.criu.JVMRestoreException: The JVM could not enable all the restore options specified
00:33:44   [OUT] 	at java.base/openj9.internal.criu.InternalCRIUSupport.checkpointJVMImpl(Native Method)
00:33:44   [OUT] 	at java.base/openj9.internal.criu.InternalCRIUSupport.checkpointJVM(InternalCRIUSupport.java:941)
00:33:44   [OUT] 	at openj9.criu/org.eclipse.openj9.criu.CRIUSupport.checkpointJVM(CRIUSupport.java:513)
00:33:44   [OUT] 	... 3 more
00:33:44   [OUT] Warn  (criu/kerndat.c:857): Stale /run/criu.kdat file
00:33:44   [OUT] Error (criu/util.c:608): exited, status=3
00:33:44   [OUT] Error (criu/util.c:608): exited, status=3
00:33:44   [OUT] Write 4294967295 to /proc/self/loginuid failed: Operation not permittedWarn  (criu/net.c:2910): Unable to get socket network namespace
00:33:44   [OUT] Warn  (criu/net.c:2910): Unable to get tun network namespace
00:33:44   [OUT] Warn  (criu/sk-unix.c:229): unix: Unable to open a socket file: Bad address
00:33:44   [OUT] Error (criu/net.c:3195): Unable create a network namespace: Operation not permitted
00:33:44   [OUT] Warn  (criu/net.c:3247): NSID isn't reported for network links
00:33:44   [OUT] Warn  (criu/net.c:2910): Unable to get socket network namespace
00:33:44   [OUT] Error (criu/util.c:1395): rlimit: Can't setup RLIMIT_NOFILE for self: Operation not permitted
00:33:44   [OUT] Error (criu/util.c:457): Can't read link of fd -404: No such file or directory
00:33:44   [OUT] Error (criu/protobuf.c:75): Unexpected EOF on (null)
00:33:44   [OUT] Removed test output files
00:33:44   [OUT] finished script
00:33:44  >> Success condition was not found: [Output match: Killed]
00:33:44  >> Failure condition was not found: [Output match: failed properties test]
00:33:44  >> Required condition was found: [Output match: Pre-checkpoint]
00:33:44  >> Success condition was not found: [Output match: Post-checkpoint]
00:33:44  >> Failure condition was not found: [Output match: CRIU is not enabled]
00:33:44  >> Failure condition was found: [Output match: Operation not permitted]
00:33:44  >> Success condition was not found: [Output match: Thread pid mismatch]
00:33:44  >> Success condition was not found: [Output match: do not match expected]
00:33:44  >> Success condition was not found: [Output match: Unable to create a thread:]
00:33:44  >> Failure condition was not found: [Output match: Could not dump the JVM processes, err=-70]
00:33:44  >> Failure condition was not found: [Output match: User requested Java dump using]

[ymanton]

We can lump them all together as failures caused by using a newer CRIU; I'll investigate them together.

I can't tell what the problem with the first test is, there's no obvious error message, but the second looks like failure to restore, but I don't know why just this test fails with permission issues when the other CRIU tests fail pass.

[pshipton]

https://openj9-jenkins.osuosl.org/job/Test_openjdk22_j9_sanity.functional_x86-64_linux_Nightly_testList_0/12 - ub16x64j95

[pshipton]

https://openj9-jenkins.osuosl.org/job/Test_openjdk22_j9_sanity.functional_x86-64_linux_Nightly_testList_1/13 - ub16x64j96
https://openj9-jenkins.osuosl.org/job/Test_openjdk21_j9_sanity.functional_x86-64_linux_Nightly_testList_1/116 - ub16x64j98
https://openj9-jenkins.osuosl.org/job/Test_openjdk22_j9_sanity.functional_x86-64_linux_Nightly_testList_0/14/ - ub16x64j96
https://openj9-jenkins.osuosl.org/job/Test_openjdk22_j9_sanity.functional_x86-64_linux_Release_testList_1/1/ - ub16x64j96
https://openj9-jenkins.osuosl.org/job/Test_openjdk21_j9_sanity.functional_x86-64_linux_Nightly_testList_0/117/ - ub16x64j96
https://openj9-jenkins.osuosl.org/job/Test_openjdk21_j9_sanity.functional_x86-64_linux_Nightly_testList_1/117/ - ub16x64j98