GithubHelp home page GithubHelp logo

jetbrains-research / testspark Goto Github PK

View Code? Open in Web Editor NEW
30.0 30.0 5.0 841.31 MB

TestSpark - a plugin for generating unit tests. TestSpark natively integrates different AI-based test generation tools and techniques in the IDE. Started by SERG TU Delft. Currently under implementation by JetBrains Research (ICTL) for research purposes.

License: MIT License

Java 1.08% Kotlin 98.90% Shell 0.03%

testspark's People

Contributors

actions-user avatar apanichella avatar arksap2002 avatar bolkhod avatar jzelenjak avatar kirilvasilev16 avatar lyuben-todorov avatar martinmladenov avatar mitchellolsthoorn avatar nikolaisviridov avatar pderakhshanfar avatar sergeydatskiv avatar vladislav0art avatar vovak avatar

Stargazers

avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar

Watchers

avatar

Forkers

vladislav0art nekitivlev kuvaevm

testspark's Issues

`IllegalArgumentException` on the generation using LLM test generation with JetBrains AI Assistant

Describe the bug
Once I opened TestSpark modal window, selected LLM as a generator, then selected AI Assistant JetBrains without modifications of chat model (i.e. leaving GPT-4), I received an error (see trace below), however it was not expected.

To Reproduce
Steps to reproduce the behavior:

  1. Select LLM as a test generator in the modal window.
  2. Select AI Assistant JetBrains.
  3. Start test generation.
  4. See error.

Expected behavior
Successful test generation. Seams like the default chat model is set to an empty string, although the GPT-4 is rendered on UI.

Stack trace

java.lang.IllegalArgumentException:  is not available option for profile. Available options are: [GPT-4, GPT-4-32k, GPT-4-Turbo, ChatGPT, ChatGPT-16k]
	at org.jetbrains.research.testSpark.grazie.TestGeneration.generate(TestGeneration.kt:31)
	at org.jetbrains.research.grazie.Request$request$1.invokeSuspend(Request.kt:20)
	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
	at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:108)
	at kotlinx.coroutines.EventLoopImplBase.processNextEvent(EventLoop.common.kt:280)
	at kotlinx.coroutines.BlockingCoroutine.joinBlocking(Builders.kt:85)
	at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking(Builders.kt:59)
	at kotlinx.coroutines.BuildersKt.runBlocking(Unknown Source)
	at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking$default(Builders.kt:38)
	at kotlinx.coroutines.BuildersKt.runBlocking$default(Unknown Source)
	at org.jetbrains.research.grazie.Request.request(Request.kt:19)
	at org.jetbrains.research.testspark.tools.llm.generation.grazie.GrazieRequestManager.send(GrazieRequestManager.kt:31)
	at org.jetbrains.research.testspark.tools.llm.generation.RequestManager.request(RequestManager.kt:43)
	at org.jetbrains.research.testspark.tools.llm.generation.RequestManager.request$default(RequestManager.kt:30)
	at org.jetbrains.research.testspark.services.LLMChatService.testGenerationRequest(LLMChatService.kt:47)
	at org.jetbrains.research.testspark.tools.llm.generation.LLMProcessManager.runTestGenerator(LLMProcessManager.kt:112)
	at org.jetbrains.research.testspark.tools.Pipeline$runTestGeneration$1.run(Pipeline.kt:66)
	at com.intellij.openapi.progress.impl.CoreProgressManager.startTask(CoreProgressManager.java:428)
	at com.intellij.openapi.progress.impl.ProgressManagerImpl.startTask(ProgressManagerImpl.java:115)
	at com.intellij.openapi.progress.impl.CoreProgressManager.lambda$runProcessWithProgressAsynchronously$6(CoreProgressManager.java:478)
	at com.intellij.openapi.progress.impl.ProgressRunner.lambda$submit$4(ProgressRunner.java:251)
	at com.intellij.openapi.progress.ProgressManager.lambda$runProcess$0(ProgressManager.java:71)
	at com.intellij.openapi.progress.impl.CoreProgressManager.lambda$runProcess$2(CoreProgressManager.java:186)
	at com.intellij.openapi.progress.impl.CoreProgressManager.lambda$executeProcessUnderProgress$13(CoreProgressManager.java:604)
	at com.intellij.openapi.progress.impl.CoreProgressManager.registerIndicatorAndRun(CoreProgressManager.java:679)
	at com.intellij.openapi.progress.impl.CoreProgressManager.computeUnderProgress(CoreProgressManager.java:635)
	at com.intellij.openapi.progress.impl.CoreProgressManager.executeProcessUnderProgress(CoreProgressManager.java:603)
	at com.intellij.openapi.progress.impl.ProgressManagerImpl.executeProcessUnderProgress(ProgressManagerImpl.java:61)
	at com.intellij.openapi.progress.impl.CoreProgressManager.runProcess(CoreProgressManager.java:173)
	at com.intellij.openapi.progress.ProgressManager.runProcess(ProgressManager.java:71)
	at com.intellij.openapi.progress.impl.ProgressRunner.lambda$submit$5(ProgressRunner.java:251)
	at com.intellij.openapi.progress.impl.ProgressRunner.lambda$launchTask$18(ProgressRunner.java:465)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.util.concurrent.Executors$PrivilegedThreadFactory$1$1.run(Executors.java:702)
	at java.base/java.util.concurrent.Executors$PrivilegedThreadFactory$1$1.run(Executors.java:699)
	at java.base/java.security.AccessController.doPrivileged(AccessController.java:399)
	at java.base/java.util.concurrent.Executors$PrivilegedThreadFactory$1.run(Executors.java:699)
	at java.base/java.lang.Thread.run(Thread.java:833)
Screenshot 2024-02-16 at 15 49 45

`NullPointerException` occurs for test generation of a line using EvoSuite

Describe the bug

I tried to generated tests for the following class:

public class Calc {
    public int sum(int a, int b) {
        return a + b; // line 3
    }

    public int multiply(int a, int b) {
        return a * b;
    }
}

I launched generation for the line 3 using EvoSuite with DYNOMOSA and it resulted in the following Exception:

java.lang.NullPointerException
	at org.jetbrains.research.testspark.services.TestsExecutionResultService.getError(TestsExecutionResultService.kt:75)
	at org.jetbrains.research.testspark.display.TestCasePanelFactory.getError(TestCasePanelFactory.kt:547)
	at org.jetbrains.research.testspark.display.TopButtonsPanelFactory.updateTopLabels(TopButtonsPanelFactory.kt:62)
	at org.jetbrains.research.testspark.services.TestCaseDisplayService.displayTestCases(TestCaseDisplayService.kt:147)
	at org.jetbrains.research.testspark.services.ReportLockingService.receiveReport(ReportLockingService.kt:32)
	at org.jetbrains.research.testspark.tools.Display.run$lambda$0(Manager.kt:135)
	// the following entries are internals of intellij openapi:
	at com.intellij.openapi.application.TransactionGuardImpl.runWithWritingAllowed(TransactionGuardImpl.java:208)
	at com.intellij.openapi.application.TransactionGuardImpl.access$100(TransactionGuardImpl.java:21)
	at com.intellij.openapi.application.TransactionGuardImpl$1.run(TransactionGuardImpl.java:190)
	at com.intellij.openapi.application.impl.ApplicationImpl.runIntendedWriteActionOnCurrentThread(ApplicationImpl.java:861)
	at com.intellij.openapi.application.impl.ApplicationImpl$4.run(ApplicationImpl.java:478)
	at com.intellij.openapi.application.impl.FlushQueue.doRun(FlushQueue.java:79)
	at com.intellij.openapi.application.impl.FlushQueue.runNextEvent(FlushQueue.java:121)
	at com.intellij.openapi.application.impl.FlushQueue.flushNow(FlushQueue.java:41)
	at java.desktop/java.awt.event.InvocationEvent.dispatch(InvocationEvent.java:318)
	at java.desktop/java.awt.EventQueue.dispatchEventImpl(EventQueue.java:789)
	at java.desktop/java.awt.EventQueue$3.run(EventQueue.java:740)
	at java.desktop/java.awt.EventQueue$3.run(EventQueue.java:734)
	at java.base/java.security.AccessController.doPrivileged(AccessController.java:399)
	at java.base/java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:86)
	at java.desktop/java.awt.EventQueue.dispatchEvent(EventQueue.java:759)
	at com.intellij.ide.IdeEventQueue.defaultDispatchEvent(IdeEventQueue.kt:685)
	at com.intellij.ide.IdeEventQueue._dispatchEvent$lambda$10(IdeEventQueue.kt:589)
	at com.intellij.openapi.application.impl.ApplicationImpl.runWithoutImplicitRead(ApplicationImpl.java:1485)
	at com.intellij.ide.IdeEventQueue._dispatchEvent(IdeEventQueue.kt:589)
	at com.intellij.ide.IdeEventQueue.access$_dispatchEvent(IdeEventQueue.kt:67)
	at com.intellij.ide.IdeEventQueue$dispatchEvent$processEventRunnable$1$1$1.compute(IdeEventQueue.kt:369)
	at com.intellij.ide.IdeEventQueue$dispatchEvent$processEventRunnable$1$1$1.compute(IdeEventQueue.kt:368)
	at com.intellij.openapi.progress.impl.CoreProgressManager.computePrioritized(CoreProgressManager.java:787)
	at com.intellij.ide.IdeEventQueue$dispatchEvent$processEventRunnable$1$1.invoke(IdeEventQueue.kt:368)
	at com.intellij.ide.IdeEventQueue$dispatchEvent$processEventRunnable$1$1.invoke(IdeEventQueue.kt:363)
	at com.intellij.ide.IdeEventQueueKt.performActivity$lambda$1(IdeEventQueue.kt:992)
	at com.intellij.openapi.application.TransactionGuardImpl.performActivity(TransactionGuardImpl.java:105)
	at com.intellij.ide.IdeEventQueueKt.performActivity(IdeEventQueue.kt:992)
	at com.intellij.ide.IdeEventQueue.dispatchEvent$lambda$7(IdeEventQueue.kt:363)
	at com.intellij.openapi.application.impl.ApplicationImpl.runIntendedWriteActionOnCurrentThread(ApplicationImpl.java:861)
	at com.intellij.ide.IdeEventQueue.dispatchEvent(IdeEventQueue.kt:405)

To Reproduce

Steps to reproduce the behavior:

  1. Copy the class to be tested above.
  2. Right click on the line 3 and select TestSpark.
  3. Select EvoSuite and the 3rd line for testing.
  4. Select DYNOMOSA (the problem might persist for other algorithms) and start generation.

Caution: not every execution results in this exception. I managed to reproduce it 3/6 times.

Expected behavior

Tests for a line should be generated with no errors.

Screenshots
1
2

3-1
3-2

Additional context

It might be specific for Windows 10, I did not test it on Unix-based systems.

"Live Learning Model" instead of "Large Language Model" in comments

Describe the bug
There several files where there is an incorrect comment "Live Learning Model" instead of "Large Language Model":

  1. CONTRIBUTING.md
  2. OpenAIRequestManager

The same thing for one of SettingsArguments.kt file with a excerpt "Longest Lasting Message".

Expected behavior
Correct comment with "Large Language Model" excerpt.

Number of passed tests is not updated properly on tests deletion

Describe the bug

After test generation and execution (i.e. after click on the "Run all" button) the number of passed tests is N/N. If I delete any of the test instances the total number of tests will be updated, but the number of passed tests will not, i.e. it will display as N/N-1 near the "Passed" label.

To Reproduce
Steps to reproduce the behavior:

  1. Generate N tests
  2. Click "Run all" button (assuming it result in N/N passed tests)
  3. Delete any test instance -> you get N/N-1 passed tests

Expected behavior
Number of passed tests should not exceed the number of total test cases.

Screenshots

Here I generated 8 tests and then deleted 2 of them:

1

Next I delete 2 more tests resulting in 4 remaining tests:

2

Additional context
Execution platform: Windows 10.

`NullPointerException` on "Run all" button click

Describe the bug
After test generation and a click of "Run all" button the following NullPointerException occurred:

java.lang.NullPointerException
	at org.jetbrains.research.testspark.display.TestCasePanelFactory.updateUI(TestCasePanelFactory.kt:335)
	at org.jetbrains.research.testspark.display.TestCasePanelFactory.runTest$lambda$13(TestCasePanelFactory.kt:474)
	at java.desktop/java.awt.event.InvocationEvent.dispatch(InvocationEvent.java:318)
	at java.desktop/java.awt.EventQueue.dispatchEventImpl(EventQueue.java:789)
	at java.desktop/java.awt.EventQueue$3.run(EventQueue.java:740)
	at java.desktop/java.awt.EventQueue$3.run(EventQueue.java:734)
	at java.base/java.security.AccessController.doPrivileged(AccessController.java:399)
	at java.base/java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:86)
	at java.desktop/java.awt.EventQueue.dispatchEvent(EventQueue.java:759)
	at com.intellij.ide.IdeEventQueue.defaultDispatchEvent(IdeEventQueue.kt:685)
	at com.intellij.ide.IdeEventQueue._dispatchEvent$lambda$10(IdeEventQueue.kt:589)
	at com.intellij.openapi.application.impl.ApplicationImpl.runWithoutImplicitRead(ApplicationImpl.java:1485)
	at com.intellij.ide.IdeEventQueue._dispatchEvent(IdeEventQueue.kt:589)
	at com.intellij.ide.IdeEventQueue.access$_dispatchEvent(IdeEventQueue.kt:67)
	at com.intellij.ide.IdeEventQueue$dispatchEvent$processEventRunnable$1$1$1.compute(IdeEventQueue.kt:369)
	at com.intellij.ide.IdeEventQueue$dispatchEvent$processEventRunnable$1$1$1.compute(IdeEventQueue.kt:368)
	at com.intellij.openapi.progress.impl.CoreProgressManager.computePrioritized(CoreProgressManager.java:787)
	at com.intellij.ide.IdeEventQueue$dispatchEvent$processEventRunnable$1$1.invoke(IdeEventQueue.kt:368)
	at com.intellij.ide.IdeEventQueue$dispatchEvent$processEventRunnable$1$1.invoke(IdeEventQueue.kt:363)
	at com.intellij.ide.IdeEventQueueKt.performActivity$lambda$1(IdeEventQueue.kt:992)
	at com.intellij.openapi.application.TransactionGuardImpl.performActivity(TransactionGuardImpl.java:105)
	at com.intellij.ide.IdeEventQueueKt.performActivity(IdeEventQueue.kt:992)
	at com.intellij.ide.IdeEventQueue.dispatchEvent$lambda$7(IdeEventQueue.kt:363)
	at com.intellij.openapi.application.impl.ApplicationImpl.runIntendedWriteActionOnCurrentThread(ApplicationImpl.java:861)
	at com.intellij.ide.IdeEventQueue.dispatchEvent(IdeEventQueue.kt:405)
	at java.desktop/java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:207)
	at java.desktop/java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:128)
	at java.desktop/java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:117)
	at java.desktop/java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:113)
	at java.desktop/java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:105)
	at java.desktop/java.awt.EventDispatchThread.run(EventDispatchThread.java:92)

To Reproduce
The following steps do not insure reproduction of the above exception but these are the steps I took in order to raise the exception.
Steps to reproduce the behavior:

  1. Generate tests for a class.
  2. Click the "Run all" button inside TestSpark sidebar.
  3. Check whether the exception is raised.

Expected behavior
Tests should be run with no errors.

Screenshots
Once I encounter the same error again I will update the issue.

Additional context
The described above steps do not guarantee reproduction of the bug.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.