LLM could not generate any tests for this class. Asking the AI assistance to fix its mistake. about testspark

[pderakhshanfar]

Hello,

We have a feedback cycle that in case of compilation issues in the tests generated by the model, sends the compilation issue to the model for fixing the problem. Any time that the plugin shows the LLM warning, it actually detects a compilation issue in tests generated by LLM. By default, the plugin stops this cycle after three unsuccessful iterations (the LLM error message).

Possible solutions for fixing this issue:

1. Simply rerun the process.
2. Try to generate tests for smaller units (such as each method in this class), separately.
3. Improve the prompt template in the settings.
4. Increase the number of iterations of the LLM feedback cycle in the settings.
5. Use another model for test generation.

Also, if the class that you want to test is openly available, we will be happy to try it ourselves and see if we can find an anomaly that can be improved from our side.

[NCCurry30]

Thanks a lot for the answer. I am using TestSpark version 0.1.8 on the following project: commons-cli
While running TestSpark against all the methods in the project, I encountered issues. The error messages I received are 'prompt too long' or 'The result is invalid or uses unknown commands due to randomness and lack of guarantees from Chat GPT. Please try again.' Could you please help me identify which step might be going wrong?
Your assistance would be greatly appreciated."

[pderakhshanfar]

Thank you for your message.

I will try the LLM-based test generation on this project and let you know my findings. If I find some solutions, I will be sure to share them with you. If I find some bugs, we will fix them as soon as possible. I will try to do it quickly, but it might take a few days.

[pderakhshanfar]

Hi again,

I managed to reproduce the LLM could not generate any tests error when I tried to generate tests for class org.apache.commons.cli.Util. According to my debugging, this is an issue from our default prompt template. Can you please try this solution and let me know about the result:

• Go to LLM settings (using the highlighted button in the attached photo.
  

• Change the Class prompt in the prompt editor to:

Generate unit tests in $LANGUAGE for $NAME to achieve 100% line coverage for this class.
Dont use @Before and @After test methods.
Make tests as atomic as possible.
All tests should be for $TESTING_PLATFORM.
In case of mocking, use $MOCKING_FRAMEWORK. But, do not use mocking for all tests.
Name all methods according to the template - [MethodUnderTest][Scenario]Test, and use only English letters.
The source code of class under test is as follows:
$CODE
Here are some information about other methods and classes used by the class under test. Only use them for creating objects, not your own ideas.
$METHODS
$POLYMORPHISM

• Generate tests for some of the classes with no tests generated warning (e.g., generate tests for class org.apache.commons.cli.Util).
• Check if the problem is resolved. If you face the same issue or another issue, please let us know.

If this fixes the issue with the Util class, it means that we need to update our default prompt.

[NCCurry30]

Thank you for your reply.
I want to make sure the TestSpark configuration of compile. Could you please give me an example of the following two configurations.

[pderakhshanfar]

Hey,

If you can build your project with IDEA, you don't need to set anything here. For the commons-cli, I managed to make the build happen with the mvn command (as it is mentioned in the project's README). So, no need to add anything here. According to my few tries of applying TestSpark on commons-cli, if you can build the project with IDEA, TestSpark will automatically find the build command and compilation path and eventually generate tests.

Also, here's another update about the prompt being too long error: I am trying some new approaches to reduce the prompt size in cases with a big class under test. Meanwhile, in large classes, TestSpark can still generate tests for smaller units, such as methods.

[Faiz-UT]

Hello @pderakhshanfar,
I have a gradle project and having a problem running the TestSpark plugin on a Java class in it. What exactly should I put under the "Project compilation settings" to generate test cases for the selected class?

Thank you!

[pderakhshanfar]

Hey @Faiz-UT
TestSpark is fully integrated with IntelliJ IDEA builds. So, if you can build your Gradle project using IDEA (more information here), you can keep "Project compilation settings" empty. In this case (which is the default and easiest way of test generation in TestSpark), TestSpark will run the build configuration and also detect and include all of the outputs of the project for test generation.

Alternatively, you can add all of the directories containing compiled classes and commands for compiling the project in the compilation path and compilation command, respectively.

[arksap2002]

No response