helmethair-co / scalatest-junit-runner Goto Github PK

Describe the bug
When test suite fail in lifecycle methods e.g. before/after these failures are ignored and not reported properly back to JUnit which makes a dangerous situation when failed tests are actually treated as passing

To Reproduce
Steps to reproduce the behavior:

1. clone https://github.com/jotel/aborted-tetsts-example
2. run ./gradlew check
3. Build is successful, but it should not be

Expected behavior

The build should fail and all suites should be present in build/test-results/test/

Screenshots
There are four suites and five tests in the sample build but only two of both of them are reported.

Additional context
Also ignored test in FunSuite marked ignore(...) are not properly reported as disabled.

Describe the bug
When a test fails to execute and an exception is thrown a RunAborted event is fired. It is handled by reporting to Junit that the test was "aborted" but Gradle handles aborted test as SKIPPED. We should report aborted scalatest tests (at least the ones with a throwable) as failes.

To Reproduce
Steps to reproduce the behavior:
Write a test throwing exception
Expected behavior
The test/suite should fail

Describe the bug
Currently, the engine will create a custom TestDescriptor if the test suite initialization fails (ref). This descriptor will have a UniqueId of form [engine:scalatest]/[failed:<SUITE_NAME>]. However, the engine does not support the suite discovery based on this ID. This limits the ways to interact with the engine, since the consumer can use the discovery result to further limit the tests to be executed and then pass a smaller set of UniqueIdDescriptors to execute method.

To Reproduce
This branch has a test that shows the empty discovery result if the UniqueId of this synthetic TestDescriptor is used as input.

Expected behavior
The engine detects that its synthetic descriptor is used and returns a valid discovery result.

I am trying to provide a custom reporter in Gradle

From ScalaTest docs:
https://www.scalatest.org/user_guide/using_the_runner#specifyingReporters

i.e. I want to add a reporter class

-C[configs...] <reporterclass> - causes test results to be reported to an instance of the specified fully qualified Reporter class name

Or maybe its possible but I am missing something?
Thanks in advance.

Could be done similar to other scalatest plugin:

<plugin>
	<groupId>org.scalatest</groupId>
	<artifactId>scalatest-maven-plugin</artifactId>
	<configuration>
		<reporters>org.company.SampleClass</reporters>
	</configuration>
</plugin>

Library is not working if on Scala 2.13

Describe the bug
when test is defined as

class SomeTest extends Object with AnyFunSuiteLike {
  test("some test") {
  }
}

The engine fails to discover that. This pattern us useful when tests need to extend from classes which are not from scaliest library. In such case traits are used instead of actuall classes.

To Reproduce
will create a PR with test and proposed fix

Expected behavior
test should be discovered run and reported

Test suites using the BeforeAndAfterAll suite mixin does not execute the beforeAll() and afterAll() functions

Is your feature request related to a problem? Please describe.

This is not really a big problem, but JUnit 5 supports nested tests natively (which are visible when e.g. you use @Nested annotation in the Jupiter API). This results in Gradle formatting tests like

TestClassName > Top-level test name > nested test name

Additionally, tooling like IDEA understands this format and also displays nested tests in the nested fashion, making the output more readable.

Current implementation of the Scalatest runner, however, results in flat list of tests, even if they use nesting like with FreeSpec:

class ExampleSpec extends AnyFreeSpec {
  "Top-level test name" - {
    "nested test name" in {
    }
  }
}

The test above would be displayed as

ExampleSpec > Top-level test name nested test name

i.e. the full test name is obtained from its "path" elements, joined with a space. In addition to losing the nesting information (minor inconvenience), this makes tooling like IDEA display all these tests as individual tests, which is moderately inconvenient because it complicates navigation between executed test cases.

Describe the solution you'd like

It would be nice for the Scalatest runner produce tests for JUnit, exposing their nesting structure, so the above test would look like this:

ExampleSpec > Top-level test name > nested test name

in textual output and as a proper tree structure in tooling.

Describe alternatives you've considered

Obviously, do nothing - it is not a very big deal, just a convenience issue.

Additional context

N/A

Is your feature request related to a problem? Please describe.
The engine does not respect org.scalatest.TagAnnotation annotations on test classes. The tagged test class is executed as if it was not tagged at all.

Describe the solution you'd like
I would like the engine to respect these annotations and only execute classes tagged like this when the appropriate includeTags modifier is used. Conversely, the engine should not execute classes tagged like this when the tag is mentioned in an excludeTags definition.

Describe alternatives you've considered
Using org.scalatest.Tag("<tag>") is a workaround, however this only works for test functions, not for test methods and whole classes. This means I have to add this Tag object to every function I define.

Additional context
N/A

When using maven-surefire-plugin version 3.0.0-M7 tests do not run from command-line.

Steps to reproduce the behavior:

1. git clone scalatest-junit-runner
2. Changed the version of maven-surefire-plugin to 3.0.0-M7
3. Run in command-line: mvn clean test -Dtest=SomeCodeTest
4. See output: Tests run: 0, Failures: 0, Errors: 0, Skipped: 0

Expected behavior
Two tests in 'SomeCodeTest' should run and succeed.

Build/Test environment:

• Maven 3.8.6
• Scala version - 2.13.1/2.13.8
• Scalatest version: 3.2.0-M3/3.2.13
• maven-surefire-plugin version 3.0.0-M7
• Java versions 1.8 and 17

• package filters
• class filters

Currently only class discovery selector is supported. Add support for at least

• path selector
• modules selector

I've got tests using ParallelExecution trait. This plugin is not running them in parallel.

Add readme with basic usage and quick-start.

Scala 3 is not yet supported

Gradle was not supporting Scala 3 until recently. When Gradle 7.3 is released with the support, the JUnit runner should be updated to support it as well.

test:

• 2.11/2.13/etc Scala versions
• multiple scalatest versions
• example Gradle build with Groovy and Kotlin DSL
• example to show tag/filter usage

default behaviour for unittests: run all tests even after the first fail
if scalatest.junit.skip_after_fail configuration parameter is set to true all tests after the first failure will skip

When running the test for a well-defined scalatest package. The runner may sometimes yield the following error:

failed to execute tests

org.gradle.api.internal.tasks.testing.TestSuiteExecutionException: Could not complete execution for Gradle Test Executor 1.
	at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.stop(SuiteTestClassProcessor.java:63)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
	at org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:33)
	at org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:94)
	at com.sun.proxy.$Proxy2.stop(Unknown Source)
	at org.gradle.api.internal.tasks.testing.worker.TestWorker$3.run(TestWorker.java:193)
	at org.gradle.api.internal.tasks.testing.worker.TestWorker.executeAndMaintainThreadName(TestWorker.java:129)
	at org.gradle.api.internal.tasks.testing.worker.TestWorker.execute(TestWorker.java:100)
	at org.gradle.api.internal.tasks.testing.worker.TestWorker.execute(TestWorker.java:60)
	at org.gradle.process.internal.worker.child.ActionExecutionWorker.execute(ActionExecutionWorker.java:56)
	at org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:133)
	at org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:71)
	at worker.org.gradle.process.internal.worker.GradleWorkerMain.run(GradleWorkerMain.java:69)
	at worker.org.gradle.process.internal.worker.GradleWorkerMain.main(GradleWorkerMain.java:74)
Caused by: org.junit.platform.commons.JUnitException: TestEngine with ID 'scalatest' failed to discover tests
	at org.junit.platform.launcher.core.EngineDiscoveryOrchestrator.discoverEngineRoot(EngineDiscoveryOrchestrator.java:111)
	at org.junit.platform.launcher.core.EngineDiscoveryOrchestrator.discover(EngineDiscoveryOrchestrator.java:85)
	at org.junit.platform.launcher.core.DefaultLauncher.discover(DefaultLauncher.java:92)
	at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:75)
	at org.gradle.api.internal.tasks.testing.junitplatform.JUnitPlatformTestClassProcessor$CollectAllTestClassesExecutor.processAllTestClasses(JUnitPlatformTestClassProcessor.java:99)
	at org.gradle.api.internal.tasks.testing.junitplatform.JUnitPlatformTestClassProcessor$CollectAllTestClassesExecutor.access$000(JUnitPlatformTestClassProcessor.java:79)
	at org.gradle.api.internal.tasks.testing.junitplatform.JUnitPlatformTestClassProcessor.stop(JUnitPlatformTestClassProcessor.java:75)
	at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.stop(SuiteTestClassProcessor.java:61)
	... 18 more
Caused by: org.junit.platform.commons.PreconditionViolationException: The discover() method for TestEngine with ID 'scalatest' returned a cyclic graph.
	at org.junit.platform.commons.util.Preconditions.condition(Preconditions.java:296)
	at org.junit.platform.launcher.core.EngineDiscoveryResultValidator.validate(EngineDiscoveryResultValidator.java:40)
	at org.junit.platform.launcher.core.EngineDiscoveryOrchestrator.discoverEngineRoot(EngineDiscoveryOrchestrator.java:104)
	... 25 more

A clear and concise description of what the bug is.

To Reproduce

• Check out https://github.com/tribbloid/spookystuff/tree/ddf2ef2603f73090c44f4b7b616425972eef4c12
• run ./gradlew clean test

Expected behavior

Should run all the tests

In worst case it should print which part of the graph is cyclic

Screenshots

Build/Test environment:

• Build tool [e.g. Gradle, Maven] and version : Gradle 7.4.2
• Scala version : 2.12.16
• Scalatest version: 3.2.10
• other relevant libraries/versions : JUnit 5.8.2

Additional context

Describe the bug
failed assertions in future context show as passing

To Reproduce

  test("should fail future") {
    val future = Future{
      1
    }

    future.foreach{ r =>
      r should === (2)
    }
  }

Expected behavior
Should fail

Build/Test environment:

• Gradle 7.1.1
• Scala 2.13.6
• Scalatest 3.2.9
• Junit 1.7.2
• co.helmethair:scalatest-junit-runner:0.1.9

Additional context
This fails correctly in the Intellij ScalaTest runner
Workaround is to to do assertions on Await.result instead of in future.foreach

Describe the bug
When adding utility classes/objects to test source sets, the plugin tries to scan them for tests but fails with:

class co.helmethair.scalatest.runtime.Discovery cannot access a member of class ***SomeHelper$ with modifiers "private"

To Reproduce

object SomeHelper extends MockFactory {
  def something(...)
}

placed under src/test/scala/{package} causes the above to be recognized as the test and fail.

Expected behavior
Such classes should be ignored.

Screenshots
N/A

Build/Test environment:

• Gradle 7
• Scala 2.12
• Scalatest 3.2.15

Thanks!

Describe the bug
If a scala test class fails to be constructed (for e.g., a NullPointerException), it is ignored when tests are filtered with junit tags, specifically, the flag includeTags. This should instead be reported as a failure (similar behavior to what happens without a tag filter)

To Reproduce
Steps to reproduce the behavior:

1. I created a sample project with the exact same failure https://github.com/shrey-ghost/scala-tag-filter-failure-demo
2. Checkout the above project and run ./gradlew test. This will fail because 2/4 test classes have an initialization error (NullPointerException)
3. Run the tests with the tag included. 2/4 test classes are annotated with @MyNewTag. The command to run the filtered tests is ./gradlew test -DincludeTags=tags.MyNewTag. This command will pass. However, this should have failed because the class foo.bar.InvalidTestWithTag has a NullPointerException during construction and also has the tag @MyNewTag annotated.

Expected behavior
Ideally, the last step above should have failed. In the current implementation, the "failed to init" test class is just skipped. However, the failed to init class should also have run and reported a failure.

I looked at the code and I think I know where the bug is. The original scala test class can be wrapped in one of these two descriptors

1. ScalatestTestDescriptor - which includes the tags from the original test class. This is used when the construction of the test class succeeds.
2. ScalatestFailedInitDescriptor - which does NOT include the tags from the original test class. This is used when the construction of the test class fails.

Since the ScalatestFailedInitDescriptor does not have tags (returns empty tags from the super class), any classes that failed to construct, will never pass the includeTags filtering done by JUnit. I think the ScalatestFailedInitDescriptor should also have the tags from the actual scala test class just like ScalatestTestDescriptor

Screenshots

1. Failures when I run all the tests

➜ scala-tag-filter-failure-demo git:(main) ./gradlew test
Task :test

UnknownClass.Failed to load test from foo.bar.InvalidTestWithTag: 'null' FAILED
java.lang.NullPointerException
at foo.bar.InvalidTestWithTag.(InvalidTestWithTag.scala:16)
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
........

UnknownClass.Failed to load test from foo.bar.InvalidTestWithoutTag: 'null' FAILED
java.lang.NullPointerException
at foo.bar.InvalidTestWithoutTag.(InvalidTestWithoutTag.scala:14)
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
........

results: FAILURE (4 tests, 2 successes, 2 failures, 0 skipped) Report file: ***/scala-tag-filter-failure-demo/build/reports/tests/test/index.html

4 tests completed, 2 failed
There were failing tests. See the report at: file://***/scala-tag-filter-failure-demo/build/reports/tests/test/index.html

2. The task passed when I run with includeTags

➜ scala-tag-filter-failure-demo git:(main) ./gradlew test -DincludeTags=tags.MyNewTag
Task :test

Gradle Test Executor 25 STANDARD_ERROR
Apr 14, 2022 12:38:57 PM org.junit.platform.launcher.core.EngineDiscoveryOrchestrator lambda$logTestDescriptorExclusionReasons$7
INFO: 0 containers and 3 tests were excluded because tags do not match tag expression(s): [tags.MyNewTag]
results: SUCCESS (1 tests, 1 successes, 0 failures, 0 skipped) Report file: /Users/shrey/github/scala-tag-filter-failure-demo/build/reports/tests/test/index.html

BUILD SUCCESSFUL in 1s
3 actionable tasks: 1 executed, 2 up-to-date

We can clearly see (INFO line) that 3 test classes were skipped because they do not match tag expression. Ideally, only two should have been skipped and we should have run two test classes (out of which one will pass and one will fail)

Build/Test environment:

• Gradle 7.1
• Scala 2.12.x
• Scalatest 2.12:3.0.8
• junit 1.7.1
• full details in the build.gradle file of the repro project

Let me know if you guys have the time to work on this or else I am happy to provide a patch for this (using the bug description I put above)

Describe the bug
Certain nested Scala classes break Java's classname convention/reflection

To Reproduce
If a similar structure as the following is on the test classpath the runner breaks during discovery (as well as junit-jupiter runner) with a MalformedClassName exception. The malformed classname will be package.Outer$Inner$CaseClass$

object Outer {
 object Inner {
  val variable = "value"
  case class CaseClass(a: String)
  }
}

case class HistoryRow(
Expected behavior
A problematic class on the classpath should not cause any problems.

Build/Test environment:

• Build tool: Gradle
• Scala: 2.11.12
• Scalatest: 3.2.0

Using Gradle Test Distribution Enterprise feature is not possible with the engine.

Given a build like this one:

plugins {
    id 'scala'
    id 'java-library'
}

repositories {
    mavenCentral()
}

dependencies {
    implementation "org.scala-lang:scala-library:2.12.8"
    testImplementation "org.scalatest:scalatest_2.12:3.2.0"
    testImplementation "org.scalatest:scalatest-flatspec_2.12:3.2.0"
    testImplementation "org.junit.platform:junit-platform-launcher:1.6.0"
    testRuntimeOnly "org.junit.platform:junit-platform-engine:1.6.0"
    testRuntimeOnly "co.helmethair:scalatest-junit-runner:0.1.11"
}

test {
    useJUnitPlatform {
        includeEngines 'scalatest'
        testLogging {
            events("passed", "skipped", "failed")
        }
    }
}

and a simple test:

class SomeCodeTest extends AnyFunSpec with Matchers {
  describe("someLibraryMethod") {
    it("always true") {
      def library = new Library()
      assert(library.someLibraryMethod())
    }
  }
}

The JUnit platform integration works as expected. However, when I enable test distribution, the tests are not executed

Session 2 opened on remote executor <executor id>
Partition 1 started on remote executor <executor id>
Partition 1 finished on remote executor<executor id> (passed = true, retries = 0)
Receiving output file <folder>/scala-testing/lib/build/test-distribution-outputs/test/test-results18149238746365984750/test-results2 from remote executor <executor id>
Finished receiving output file <folder>/scala-testing/lib/build/test-distribution-outputs/test/test-results18149238746365984750/test-results2 with size 111 bytes from remote executor <executor id>
Session 2 closed on remote executor <executor id>

The build is reported as successful and no tests are executed

From conversation with @marcphilip , he pointed out that the following might be happening:

It seems like the limitation is because scalatest-junit-runner only handles ClassSelectors is not able to run tests selected via unique ID: https://github.com/helmethair-co/scalatest-junit-runner/blob/f2de1f0cd4fd8587b13ae[…]f974/src/main/java/co/helmethair/scalatest/ScalatestEngine.java

It would need to check for UniqueIdSelectors in EngineDiscoveryRequest and map those to classes as well.

Unfortunately, because Test Distribution is an Enterprise issue, it is difficult to provide a reproducible

Describe the bug
One of my suites throws an exception in beforeAll method. To my surprise, this problem is not reported back to gradle and build finishes with a success.

Debugging shows that this code in Executor.java is triggered after an exception:

            } else {
                reporter.apply(runAborted(args.tracker().nextOrdinal(), e, Resources.bigProblems(e), scalasuite));
                if (!NonFatal$.MODULE$.apply(e)) {
                    throw e;
                }
            }

In my case, !NonFatal$.MODULE$.apply(e) is evaluated to false.

To Reproduce

1. Create a gradle project that uses scala + scalatest + scalatest-junit-runner.
2. Create a suite that throws RuntimeException in beforeAll()
3. Run gradle test. Build should finish successfully.

Expected behavior
Build fails with a message about aborted runs.

Build/Test environment:

• Gradle 6.6
• Scala version 2.13.3
• Scalatest version 3.2.0
• scalatest-junit-runner 0.1.8

#91