[Question] Can Jdk 21 's Vector API improve inference performance under single threads? about llama2.java

[neoremind]

Thanks for looking at the fun project. Yes, Vector API help Java gain notable performance benefit, please check out Performance Evaluation section for more details. The minimum required is Java 17+, and I tested against Java 21, the result is almost the same.

[WackyGem]

Thanks for looking at the fun project. Yes, Vector API help Java gain notable performance benefit, please check out Performance Evaluation section for more details. The minimum required is Java 17+, and I tested against Java 21, the result is almost the same.

This is an exciting result. Thanks for sharing, the performance is almost equal to C in single thread. Is there any direction to improve multithreading performance in JVM environment? As of now running a 7B model is still a bit of a struggle.

[neoremind]

As you can see running Llama2 7B, Java is as good as C with the help of vectorization, particularly in multi-threaded scenarios. But for baby llama stories series model, my guess is that Java's multi-thread context switch reduces the benefit gained through vectorization (may need further evaluation to confirm), so there is obvious performance gap for the baby models.