scalameta / munit

Scala testing library with actionable errors and extensible APIs

scalameta

User documentation

Head over to the user docs to learn more about the project and its roadmap.

Tutorial

If you'd like to find out how to use scalameta, see this tutorial.

Team

The current maintainers (people who can merge pull requests) are:

• David Dudson - @DavidDudson
• Ólafur Páll Geirsson - @olafurpg
• Krzysztof Bochenek - @kpbochenek
• Mikhail Mutcianko - @mutcianm
• Max Ovsiankin - @maxov
• Gabriele Petronella - @gabro
• Denys Shabalin - @densh

An up-to-date list of contributors is available here: https://github.com/scalameta/scalameta/graphs/contributors.

munit's Issues

Loosen assertEquals type constraint to allow subtyping

Context: https://gitter.im/scalameta/munit?at=5e36e5cadc52c34ace3fd552

Currently, the following code does not typecheck

assertEquals(List(1).headOption, Some(1))
// Cannot prove that Option[Int] =:= Some[Int]

We might want to allow the right-hand side (expected value) to be a subtype of the right hand side

assertEquals(List(1).headOption, Some(1)) // OK
assertEquals(Some(1), List(1).headOption) // ERROR: Cannot prove that Option[Int] <:< Some[Int]

The reasoning being that the expected value is often a hardcoded expression like "Some(1)" while the obtained value comes from a generic method implementation.

MUnit only reporting a handful of test results

I have converted my test suites from ScalaTest to MUnit, and for some reason MUnit only appears to be reporting a subset of my expected test results. At first I only had a single test suite, in a large file of several hundred lines, and it was reporting only 3 results. I have since moved my tests into several, smaller suites, and now it only reports results from the very last suite.

Here is some sample output, from SBT:

sbt:mymedialog-mediadb> test
[info] Compiling 7 Scala sources to /home/ryan/Projects/mymedialog-mediadb/target/scala-2.13/test-classes ...
io.medialog.mediadb.NewEditionRouteSuite:
io.medialog.mediadb.NameRouteSuite:
io.medialog.mediadb.RenameRouteSuite:
io.medialog.mediadb.ItemRouteSuite:
io.medialog.mediadb.ModifyLocalesRouteSuite:
  + POST -> /[type]/[id]/names/modify-locales should fail on an invalid type string 0.897s
  + POST -> /[type]/[id]/editions/new should fail on an invalid type string 0.914s
  + POST -> /[type]/[id]/names/rename should fail on an invalid type string 0.917s
  + GET -> /[type]/[id] should fail on an invalid type string 0.919s
  + POST -> /[type]/[id]/names should fail on an invalid type string 0.925s
  + POST -> /new should add unique, new items only 0.366s
[info] Passed: Total 6, Failed 0, Errors 0, Passed 6
[success] Total time: 9 s, completed May 30, 2020, 4:31:30 PM

As you can see, the first 4 suites look almost as if they are being skipped over, but the final suite is the only one that appears to get ran at all.

The tests for all of the above suites have the following in common:

• All use the same base suite, a combination of ScalaCheckSuite with some custom configuration and common functions (some scalacheck generators, HTTP4S DSLs and implicits, mainly)
• All tests are ScalaCheck property tests using the property function
• All tests are 100% synchronous

The code in question is on a private repository. If it helps to resolve the issue, I can grant temporary access to the repository to a project maintainer for MUnit (it's a personal project anyway). I can also share more specific code samples as needed, but the tests are pretty deeply tied into my domain logic so I can only share so much at a time.

EDIT: Some additional details:

• Scala 2.13.2
• VS Code & Metals w/ Bloop on WSL Ubuntu 20.04
• MUnit 0.7.7
• GraalVM 20.1.0 - JDK 11

discussion on assertEquals

I know a while back the semantics of assertEquals changed... and broke 100s of my test cases. :-)

What was the reason for this change?

If I do:

val a = "foo"
val b = "foo"
a == b  // true
a.equals(b) // true

For all purposes a equals b, so why does assertEquals(a,b) return false? Now I need to do: assert(a==b), which works, but I'm not clear.

Unusual behavior with ScalaCheckSuite and Gen.choose

Well here's a weird one. When using ScalaCheckSuite the Gen.choose constructor seems to choose only one value per test run. It works properly with org.scalacheck.Properties so it seems to be an munit issue.

import munit.ScalaCheckSuite
import org.scalacheck.Prop._
import org.scalacheck.Gen

class HmmSuite extends ScalaCheckSuite {

  override def scalaCheckTestParameters =
    super.scalaCheckTestParameters
         .withMinSuccessfulTests(10)

  // odd behavior
  property("choose") {
    forAll(Gen.choose(0,Int.MaxValue)) { n =>
      println(n)
      true
    }
  }

  // seems ok?
  property("posNum") {
    forAll(Gen.posNum[Int]) { n =>
      println(n)
      true
    }
  }

}

Test output

HmmSuite:
1644645754
1644645754
1644645754
1644645754
1644645754
1644645754
1644645754
1644645754
1644645754
1644645754
OK, passed 10 tests.
  + choose 0.137s
1
8
8
8
28
28
8
18
28
68
OK, passed 10 tests.
  + posNum 0.003s
Execution took 0.14s
2 tests, 2 passed
All tests in HmmSuite passed

Make Fixture a plain class instead of an inner class of Suite

I would love to be able to create Fixture instances outside of a test suite; alas, it is an inner class of Suite.

I currently use munit to create a CLI tool which executes multiple "tests" on a given entity, and that entity is constructed dynamically according to user input:

// would love this to extend from munit.Fixture
trait MyFixture {
  def createEntity(): Entity

  def beforeAll(): Unit = {}
  // and all the rest
}

abstract class MySuite(fixture: MyFixture) extends munit.Suite {

  protected var entity: Entity = _

  override def beforeAll(): Unit = {
    super.beforeAll()
    fixture.beforeAll()
    entity = createEntity()
  }
}

class MyRunner(cls: Class[_ <: MySuite], newInstance: () => Suite) extends MUnitRunner(cls, newInstance) {
  def this(fixture: MyFixture, cls: Class[_ <: MySuite]) = {
    this(cls, MyUtils.findCtr(cls).newInstance(fixture))
  }
}

object MyApp extends App {
  val input = ???
  val fixture = generateFixture(input)
  val suites = findSuites(input)

  val junit = new JUnitCore
  junit.addListener(new TextListener(System.out))
  suites.foreach { suite =>
    junit.run(new MyRunner(fixture, suite))
  }
}

The reason I'm constructing the entity via fixtures while executing the tests (and not outside of the execution) is because its creation is rather complex and requires both resource initialization and mocking (via mockito).

I hope it's a clear enough example, but if not just let me know and I'll further explain :)

Shade sbt-munit dependencies

The upgrade in scalameta/mdoc#276 reveals that sbt-munit is incompatible with certain versions of Scala.js bundler. We should shadow sbt-munit dependencies so it doesn't conflict with other plugins.

Please publish Dotty versions to repo

It would be great if Dotty versions of munit could be published to the maven repo.

Currently munit is one of the only testing frameworks for Dotty, so it's important in that community to have a current version. I've been maintaining and publishing a Dotty build in my repo, but an official release would be super. I'm currently built off of Dotty 0.24-RC1 against JDK11.

Formatting Enhancement for test output

This is a minor thing, but IMO it would make test output look nicer if the details were indented.
Today:

co.blocke.dotty_reflection.ScalaTasty:
+ reflect basic Tasty class with union 1.714s
+ create basic Tasty class 0.002s
co.blocke.dotty_reflection.JavaNonTasty:
+ reflect basic with capture 0.078s
+ create Java object 0.004s
+ Verify Java primitives 0.052s

Proposed:

co.blocke.dotty_reflection.ScalaTasty:
   + reflect basic Tasty class with union 1.714s
   + create basic Tasty class 0.002s
co.blocke.dotty_reflection.JavaNonTasty:
   + reflect basic with capture 0.078s
   + create Java object 0.004s
   + Verify Java primitives 0.052s

This would let your eye catch the suite-level easier.

Raise an error when “only” tests are found in CI

I think it’s far too easy to mark a test as “only” and forget to remove the modifier when committing and pushing.

It may be a good idea to fail by default is a test is marked as “only” and isCI is true.

This can probably be made overridable in the suite, in case someone wants to turn it off.

Support JUnit 5

From https://contributors.scala-lang.org/t/announcing-munit-a-new-scala-testing-library/4007/11?u=olafurpg

It does not currently support JUnit 5, at least not with gradle – trying to run with org.junit.jupiter:junit-jupiter-engine doesn’t work, and instead the “vintage” engine has to be used.

I'm not so familiar with what is needed to make this work but I think it's worth looking into.

Allow ad-hoc querying of test reports

Follow up from scalameta/metals#1373 (comment)

We should allow some way to run ad-hoc queries against the test reports, for example to see full straces and messages of exceptions.

Update build to support Dotty 0.23.0-RC1

This is a tracking issue. Dotty 0.23.0-RC1 has been tagged and is due to drop imminently. There are some minor code changes required to support the move from 0.22 to 0.23. I have a PR ready to submit as soon as it drops. This issue is to track and raise awareness for this change, and can be closed if/when the PR is merged.

Scala.js 1.0.1 support

I am trying to use MUnit with Scala.js 1.0.1 and Scala 2.13, but the linker is encountering errors:

$ sbt test
[...]
[info] Fast optimizing /tmp/testtest/munit-scalajs-10/root/target/scala-2.13/root-test-fastopt.js
[error] munit.internal.JSFs$ needs to be imported from module 'fs' but module support is disabled.
[error] munit.internal.JSPath$ needs to be imported from module 'path' but module support is disabled.
[error] There were module imports without fallback to global variables, but module support is disabled.
[error] To enable module support, set `scalaJSLinkerConfig ~= (_.withModuleKind(ModuleKind.CommonJSModule))`.
[error] (root / Test / fastOptJS) There were module imports without fallback to global variables, but module support is disabled.
[error] To enable module support, set `scalaJSLinkerConfig ~= (_.withModuleKind(ModuleKind.CommonJSModule))`.

Here is a minimal sbt project exhibiting the faulty behaviour.

I have also tried it with Bloop (using scalacenter/bloop#1234) and am getting the following errors:

[E] org.junit.runner.Runner (of kind Class) is not a valid interface implemented by munit.MUnitRunner (of kind Class)
[E]   called from munit.MUnitRunner
[E]   called from private munit.internal.PlatformCompat$.$anonfun$newRunner$1(scala.scalajs.reflect.InstantiatableClass)munit.MUnitRunner
[E]   called from munit.internal.PlatformCompat$.newRunner(sbt.testing.TaskDef,java.lang.ClassLoader)scala.Option
[E]   called from munit.internal.junitinterface.JUnitTask.execute(sbt.testing.EventHandler,[sbt.testing.Logger,scala.Function1)void
[E]   called from private org.scalajs.testing.bridge.HTMLRunner$.$anonfun$scheduleTask$1(sbt.testing.Task,org.scalajs.testing.bridge.HTMLRunner$EventCounter$Handler,org.scalajs.testing.bridge.HTMLRunner$UI$TestTask,scala.concurrent.Promise)void
[E]   called from private org.scalajs.testing.bridge.HTMLRunner$.scheduleTask(sbt.testing.Task,org.scalajs.testing.bridge.HTMLRunner$UI)scala.concurrent.Future
[E]   called from private org.scalajs.testing.bridge.HTMLRunner$.$anonfun$runTests$1(org.scalajs.testing.bridge.HTMLRunner$UI,sbt.testing.Task)scala.concurrent.Future
[E]   called from private org.scalajs.testing.bridge.HTMLRunner$.runAllTasks$1(scala.collection.Seq,org.scalajs.testing.bridge.HTMLRunner$UI)scala.concurrent.Future
[E]   called from private org.scalajs.testing.bridge.HTMLRunner$.runTests(sbt.testing.Framework,scala.collection.immutable.Seq,org.scalajs.testing.bridge.HTMLRunner$UI)scala.concurrent.Future
[E]   called from private org.scalajs.testing.bridge.HTMLRunner$.$anonfun$onLoad$10(scala.Function1,org.scalajs.testing.bridge.HTMLRunner$UI,scala.Tuple2)scala.concurrent.Future
[E]   called from private org.scalajs.testing.bridge.HTMLRunner$.onLoad(org.scalajs.testing.common.IsolatedTestSet)void
[E]   called from org.scalajs.testing.bridge.HTMLRunner$.org$scalajs$testing$bridge$HTMLRunner$$$anonfun$start$1(org.scalajs.testing.common.IsolatedTestSet)void
[E]   called from org.scalajs.testing.bridge.HTMLRunner$.start(org.scalajs.testing.common.IsolatedTestSet)void
[E]   called from org.scalajs.testing.bridge.Bridge$.start()void
[E]   called from static org.scalajs.testing.bridge.Bridge.start()void
[E]   called from core module module initializers
[E] involving instantiated classes:
[E]   munit.internal.PlatformCompat$
[E]   munit.internal.junitinterface.JUnitTask
[E]   org.scalajs.testing.bridge.HTMLRunner$
[E]   org.scalajs.testing.bridge.Bridge$

The Bloop classpath looks fine though:

"classpath" : [
	"/home/tim/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-js/scalajs-junit-test-runtime_2.13/1.0.1/scalajs-junit-test-runtime_2.13-1.0.1.jar",
	"/home/tim/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-js/scalajs-library_2.13/1.0.1/scalajs-library_2.13-1.0.1.jar",
	"/home/tim/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-js/scalajs-test-bridge_2.13/1.0.1/scalajs-test-bridge_2.13-1.0.1.jar",
	"/home/tim/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-js/scalajs-test-interface_2.13/1.0.1/scalajs-test-interface_2.13-1.0.1.jar",
	"/home/tim/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/scala-library/2.13.2/scala-library-2.13.2.jar",
	"/home/tim/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/scala-reflect/2.13.2/scala-reflect-2.13.2.jar",
	"/home/tim/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scalameta/munit_sjs1_2.13/0.7.3/munit_sjs1_2.13-0.7.3.jar",
	"/tmp/build-test-scalajs-10/bloop/example"
]

Add Scalafmt/Scalafix checks on CI

No way to «go to test definition» in IDEA

1. It would be helpful to go to test definition line in code when double clicking on the failed test name in console
2. There is no clickable source location for error (at munit.Assertions.fail(Assertions.scala:134) only)

Implement retrieving of a test status

Currently, it is not possible to retrieve a specific test status in a test suite.

Example: it could be useful to know test status in afterEach/afterAll blocks when you have to do additional actions in case of a failed test.

Tests duplication in IDEA console

IDEA renders each test three times, but executes once.

Reproduces on ScalafmtVersionSuite with IDEA 2019.3.2 (Windows / MacOS)

Tests don't compile with Dotty 0.22.0-RC1, due to problems with implicits and macros

I believe I'm using the same Dotty version, 0.22.0-RC1, as the 0.5.0 release of munit.

Consider this example test:

import scala.language.implicitConversions // same behavior if present or not.
import munit._
import munit.Location.generate // explicitly added in an attempt to fix an error below...

class MUnitExample extends FunSuite {

  test("A string has a certain length") {
    assertEquals("12345".length, 5)
  }
}

It results in the following compilation errors:

[error] -- [E007] Type Mismatch Error: .../src/test/scala/foo/bar/munit-example-test.scala:10:7
[error] 10 |  test("A string has a certain length") {
[error]    |       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[error]    |       Found:    ("A string has a certain length" : String)
[error]    |       Required: munit.TestOptions
[error]    |       Constraint(
[error]    |        uninstVars = ;
[error]    |        constrained types =
[error]    |        bounds =
[error]    |        ordering =
[error]    |       )
[error]    |       Subtype trace:
[error] -- Error: .../src/test/scala/foo/bar/munit-example-test.scala:11:35
[error] 11 |    assertEquals("12345".length, 5)
[error]    |                                   ^
[error]    |no implicit argument of type munit.Location was found for parameter loc of method assertEquals in trait Assertions.
[error]    |I found:
[error]    |
[error]    |    generate
[error]    |
[error]    |But method generate in trait LocationMacro does not match type munit.Location.
[error]    |
[error]    |The following import might make progress towards fixing the problem:
[error]    |
[error]    |  import munit.Location.generate
[error]    |
[error] -- Error:.../src/test/scala/foo/bar/munit-example-test.scala:12:3
[error] 12 |  }
[error]    |   ^
[error]    |no implicit argument of type munit.Location was found for parameter loc of method test in class FunSuite.
[error]    |I found:
[error]    |
[error]    |    generate
[error]    |
[error]    |But method generate in trait LocationMacro does not match type munit.Location.
[error]    |
[error]    |The following import might make progress towards fixing the problem:
[error]    |
[error]    |  import munit.Location.generate
[error]    |

I assume that the correct Scala 3 macro for Location is being used, but why it thinks the type is wrong I don't understand. The code for the macro looks correct.

I also don't understand enough about implicit resolution in Scala 3 to understand why the conversion to TestOptions from String isn't invoked. Perhaps it's not actually in scope, even with import munit._?

Add assertSameElements helper

It may be reasonable to have assertSameElements function that ignores elements order on collection comparison.

Now user forced to write something like this assert(diff("master").toSet == Set(f1, f2))

Unexpected blocked test execution

When writing a test case that uses assertNoDiff with two big strings, munit gets blocked and the test doesn't finish.

After having a closer look with jstack, it looks like the test is stuck at newCapacity, which is triggered by AnsiColors.filterAnsi.

"pool-1-thread-1" #19 prio=5 os_prio=31 tid=0x00007f8cfea70800 nid=0x6603 runnable [0x00007000102e9000]
   java.lang.Thread.State: RUNNABLE
	at java.lang.AbstractStringBuilder.newCapacity(AbstractStringBuilder.java:154)
	at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:125)
	at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:448)
	at java.lang.StringBuilder.append(StringBuilder.java:136)
	at munit.internal.console.AnsiColors$.filterAnsi(AnsiColors.scala:35)
	at munit.internal.difflib.Diff.<init>(Diff.scala:9)
	at munit.internal.difflib.Diffs$.assertNoDiff(Diffs.scala:20)
	at munit.Assertions.$anonfun$assertNoDiff$1(Assertions.scala:53)
	at munit.Assertions$$Lambda$194/744865283.apply$mcZ$sp(Unknown Source)
	at scala.runtime.java8.JFunction0$mcZ$sp.apply(JFunction0$mcZ$sp.scala:17)
	at munit.internal.console.StackTraces$.dropInside(StackTraces.scala:10)
	at munit.Assertions.assertNoDiff(Assertions.scala:53)
	at munit.Assertions.assertNoDiff$(Assertions.scala:47)
	at pulsar.testing.BaseSpec.assertNoDiff(BaseSuite.scala:450)
	at pulsar.NbformatSpec.$anonfun$checkNbformatRoundTrip$6(NbformatSpec.scala:16)
	at pulsar.NbformatSpec.$anonfun$checkNbformatRoundTrip$6$adapted(NbformatSpec.scala:14)
	at pulsar.NbformatSpec$$Lambda$122/2086796509.apply(Unknown Source)
	at monix.eval.Task$Map.apply(Task.scala:4541)
	at monix.eval.Task$Map.apply(Task.scala:4537)
	at monix.eval.internal.TaskRunLoop$.startFuture(TaskRunLoop.scala:550)
	at monix.eval.Task.runToFutureOpt(Task.scala:585)
	at monix.eval.Task.runToFuture(Task.scala:541)
	at pulsar.NbformatSpec.checkNbformatRoundTrip(NbformatSpec.scala:22)
	at pulsar.NbformatSpec.$anonfun$new$2(NbformatSpec.scala:30)
	at pulsar.NbformatSpec$$Lambda$68/1205689408.apply(Unknown Source)
	at munit.internal.console.StackTraces$.dropOutside(StackTraces.scala:12)
	at munit.ValueTransforms.$anonfun$munitValueTransform$3(ValueTransforms.scala:42)
	at munit.ValueTransforms$$Lambda$94/1111438620.apply(Unknown Source)
	at scala.util.Try$.apply(Try.scala:210)
	at munit.ValueTransforms.munitValueTransform(ValueTransforms.scala:42)
	at munit.ValueTransforms.munitValueTransform$(ValueTransforms.scala:27)
	at munit.FunSuite.munitValueTransform(FunSuite.scala:7)
	at munit.FunSuite.$anonfun$test$1(FunSuite.scala:32)
	at munit.FunSuite$$Lambda$64/857546750.apply(Unknown Source)
	at munit.MUnitRunner.$anonfun$runTestBody$1(MUnitRunner.scala:268)
	at munit.MUnitRunner$$Lambda$84/1125581248.apply(Unknown Source)
	at munit.internal.console.StackTraces$.dropOutside(StackTraces.scala:12)
	at munit.MUnitRunner.runTestBody(MUnitRunner.scala:264)
	at munit.MUnitRunner.runTest(MUnitRunner.scala:246)
	at munit.MUnitRunner.loop$1(MUnitRunner.scala:113)
	at munit.MUnitRunner.$anonfun$runAsyncTestsSynchronously$1(MUnitRunner.scala:113)
	at munit.MUnitRunner.$anonfun$runAsyncTestsSynchronously$1$adapted(MUnitRunner.scala:113)
	at munit.MUnitRunner$$Lambda$209/1092806825.apply(Unknown Source)
	at scala.concurrent.impl.Promise$Transformation.run(Promise.scala:433)
	at munit.Suite$$anon$1.execute(Suite.scala:25)
	at scala.concurrent.impl.Promise$Transformation.submitWithValue(Promise.scala:392)
	at scala.concurrent.impl.Promise$DefaultPromise.submitWithValue(Promise.scala:302)
	at scala.concurrent.impl.Promise$DefaultPromise.dispatchOrAddCallbacks(Promise.scala:276)
	at scala.concurrent.impl.Promise$DefaultPromise.flatMap(Promise.scala:140)
	at munit.MUnitRunner.loop$1(MUnitRunner.scala:113)
	at munit.MUnitRunner.runAsyncTestsSynchronously(MUnitRunner.scala:115)
	at munit.MUnitRunner.runAll(MUnitRunner.scala:128)
	at munit.MUnitRunner.runAsync(MUnitRunner.scala:95)
	at munit.MUnitRunner.run(MUnitRunner.scala:90)
	at munit.internal.junitinterface.JUnitRunnerWrapper.run(JUnitRunnerWrapper.java:24)
	at org.junit.runners.Suite.runChild(Suite.java:128)
	at org.junit.runners.Suite.runChild(Suite.java:27)
	at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
	at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
	at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
	at org.junit.runner.JUnitCore.run(JUnitCore.java:115)
	at munit.internal.junitinterface.JUnitTask.execute(JUnitTask.java:80)
	at sbt.ForkMain$Run.lambda$runTest$1(ForkMain.java:304)
	at sbt.ForkMain$Run$$Lambda$7/754666084.call(Unknown Source)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

I've had a look at the code and it looks like this might be a memory problem. Replacing r to be a StringBuilder instead of a String might be the solution to this problem.

Enable MiMa in CI

Now that there are some external projects have published integrations with MUnit (https://github.com/47degrees/docker-testkit-munit) we should probably enable MiMa in CI so that we don't release new versions with accidental breaking changes.

Can't resolve junit-interfaces after latest build

When I do a compile/publishLocal, munit and junit-interfaces get 2 different generated SNAPSHOT versions. For example my latest attempt:

munit: 0.5.2+3-cb874360+20200218-1135-SNAPSHOT
junit-interfaces: 0.5.2+3-cb874360+20200218-1136-SNAPSHOT

This means when I go to use munit I get a dependency failure on junit-interfaces. (I'm pulling in the munit dependency in my project, and munit is pulling in junit-interfaces, but it's looking for the same version as munit.)

These 2 artifacts should have the same version, right?

Support NO_COLOR environment variable

Currently, the only way to disable ANSI color output from MUnit is to use the cryptic combination of the testing framework flags testOnly -- -n +l. We should make it easier to disable colors in MUnit by respecting the NO_COLOR environment variable https://no-color.org/

Allow munitIgnore to include an optional explanation?

Currently override def munitIgnore = Properties.isWin does make it possible to explain why the test suite is ignored. Should we somehow allow including an optional message?

ScalaCheckSuite unexpected behavior

It seems that the ScalaCheckSuite tests are not generating different values during the forAll iterations for types that are not collections or Strings (like Int, Double or even UUID). Tests like the below

final class IntGeneratorsSuite extends ScalaCheckSuite {

  val ints: mutable.Set[Int] = mutable.Set.empty

  property("generating int") {
    forAll { i: Int =>
      ints.add(i)
      println(ints)
      true
    }
  }

  test("generated int are not all the same") {
    assert(ints.size > 1)
  }
}

fail with the current master. and print like

HashSet(-66888203)
HashSet(-66888203)
HashSet(-66888203)
HashSet(-66888203)
HashSet(-66888203)
HashSet(-66888203)
HashSet(-66888203)
HashSet(-66888203)
HashSet(-66888203)
HashSet(-66888203)
HashSet(-66888203)
HashSet(-66888203)
HashSet(-66888203)
HashSet(-66888203)
HashSet(-66888203)
HashSet(-66888203)
HashSet(-66888203)
HashSet(-66888203)
HashSet(-66888203)
HashSet(-66888203)
HashSet(-66888203)
[...]

This might be related to the fact that an initial seed is always set here and scalacheck copies that into the Gen's initial seed and keeps using it to create values (see genPrms here ,forAllShrink here and startSeed here.
I'm not sure about what a possible solution could be. I played a bit with the code but I couldn't find any way to set an initial seed in the Test's params and get different values as I was expecting.

Missing release notes for 0.7.11

https://github.com/scalameta/munit/releases/tag/v0.7.11 has no info — oversight?

Replace TestOnly.{only, ignore, flag} methods with TestOnly.tag({Only, Ignore, Flag})

Would it make sense to drop the custom .ignore, .only, ..., methods in favor of always using tag(...)?

- test("foo".ignore)
+ test("foo".tag(Ignore))

The motivation for this change would be to have only one way to do things.

Consider compat lib for seamless ScalaTest migration

Hi @olafurpg ,

I just found about Munit at the ScalaLove conf and decided to give it a go in one of my projects using ScalaTest and WordSpec so far.

In due course, I have found a nice way to migrate to MUnit without rewriting a single line of my tests, except of course of changing class dependency to munit.FunSuite. That would make it easier for many people to try MUnit first and then eventually switch.

The resulting compatibility trait and an example test is presented here https://gist.github.com/arturopala/4cfd20de97006031329a946f08c54f20

Do you think it would make sense to have such a compatibility tool in a separate project?

Cheers, Artur

Support suite local fixtures without mutable state

Currently, you can write test-local fixtures without mutable state but you need to resort to var and null if you want to reuse a fixture for all test cases in a suite. It would be nice to have a way to declare suite-local fixtures without mutable state.

Using wrong execution context cause unexpected exception

Using:

  override def munitTestTransforms: List[TestTransform] =
    super.munitTestTransforms :+
      new TestTransform("Print DAP traces", { test =>
        test.withBody(() =>
          test
            .body()
            .andThen {
              case Failure(exception) =>
                logDapTraces()
                exception
              case Success(value) => value
            }(someExecutionContext)
        )
      })

will cause an exception to be thrown, for example:

X tests.debug.StackFrameDapSuite.afterAll  0.002s java.lang.IllegalArgumentException: requirement failed
2020-04-15T12:00:59.8858923Z [error]     at scala.Predef$.require(Predef.scala:268)
2020-04-15T12:00:59.8859382Z [error]     at scala.concurrent.BatchingExecutor$Batch.run(BatchingExecutor.scala:55)
2020-04-15T12:00:59.8859785Z [error]     at scala.concurrent.Future$InternalCallbackExecutor$.unbatchedExecute(Future.scala:875)
2020-04-15T12:00:59.8860176Z [error]     at scala.concurrent.BatchingExecutor$Batch.blockOn(BatchingExecutor.scala:96)
2020-04-15T12:00:59.8872621Z [error]     at scala.concurrent.package$.blocking(package.scala:146)
2020-04-15T12:00:59.8873105Z [error]     at scala.concurrent.java8.FuturesConvertersImpl$CF.get(FutureConvertersImpl.scala:92)
2020-04-15T12:00:59.8873645Z [error]     at scala.meta.internal.metals.MetalsLanguageServer.cancel(MetalsLanguageServer.scala:89)
2020-04-15T12:00:59.8874081Z [error]     at scala.meta.internal.metals.MetalsLanguageServer.cancelAll(MetalsLanguageServer.scala:97)
2020-04-15T12:00:59.8874432Z [error]     at tests.BaseLspSuite.afterAll(BaseLspSuite.scala:46)
2020-04-15T12:00:59.8874769Z [error]     at munit.MUnitRunner.$anonfun$runAfterAll$3(MUnitRunner.scala:160)

To fix it we need to use munitExecutionContext. I think that can cause some confusion on the user part and it took me a while to figure it out. We should make sure that it's easy to figure out what execution context should be used.

Website content failure on build

I'm building current (as of this writing) master-branch munit against a 0.23 NIGHTLY Dotty release and getting these errors when I do a + publishLocal:

-- Error: website/node_modules/balanced-match/README.md:50:132 -----------------
50 |If the `str` contains more `a` than `b` / there are unmatched pairs, the first match that was closed will be used. For example, `{{a}` will match `['{', 'a', '']` and `{a}}` will match `['', 'a', '}']`.
   |                                                                                                                                    ^
   |                              unexpected NoSpace, expected '}}' (OutEnd)
-- Error: website/node_modules/brace-expansion/README.md:41:11 -----------------
41 |expand('{{A..C},{a..c}}')
   |           ^
   |           unexpected '..' (DotDot), expected '}}' (OutEnd)
[error] stack trace is suppressed; run last munitJVM / Compile / doc for the full output
[error] (munitJVM / Compile / doc) java.lang.reflect.InvocationTargetException
[error] Total time: 46 s, completed Mar 16, 2020, 10:43:06 AM

Support specifying test options on Dotty without enabling implicit conversions

build.sbt:

scalaVersion := "0.23.0-RC1"
libraryDependencies += "org.scalameta" %% "munit" % "0.7.1"
testFrameworks += new TestFramework("munit.Framework")

Source:

package foo

import munit.FunSuite

class Foo extends FunSuite {
  test("foo".ignore) {
    assert(true)
  }
}

Result of compiling:

[info] Compiling 1 Scala source to /Users/mpilquist/Development/oss/munit-example/target/scala-0.23/classes ...
[warn] -- Feature Warning: /Users/mpilquist/Development/oss/munit-example/src/main/scala/Test.scala:6:7
[warn] 6 |  test("foo".ignore) {
[warn]   |       ^^^^^
[warn]   |Use of implicit conversion method testOptionsFromString in trait TestOptionsConversions should be enabled
[warn]   |by adding the import clause 'import scala.language.implicitConversions'
[warn]   |or by setting the compiler option -language:implicitConversions.
[warn]   |See the Scala docs for value scala.language.implicitConversions for a discussion
[warn]   |why the feature should be explicitly enabled.
[warn] one warning found

If we change testOptionsFromString from an implicit def to an implicit Conversion[String, TestOptions], along with a shim definition of type Conversion[-A, +B] = A => B on Scala 2, we can avoid this warning.

Blocking Async Tests

It looks like the current async runner requires blocking Await.result. This also has the adverse result that async code is not testable in JS, see:

munit/munit/js/src/main/scala/munit/internal/PlatformCompat.scala

Line 19 in ee3bff5

case Some(value) =>

.

Utest works around this by providing an async interface with callbacks for the JS side, and on the JVM side will block on the full evaluation of the test tree.

Fail exception report is incorrect

I get the following exception with the code that follows:
values are not equal even if they have the same toString(): bb as,dn

Longer version:

==> X org.scalajs.testsuite.javalib.util.PropertiesSuite.string leading  0.001s munit.FailException: /Users/eric/workspace/portable-scala-java-test/testSuite/shared/src/test/scala/org/scalajs/testsuite/javalib/util/PropertiesSuite.scala:162 values are not equal even if they have the same `toString()`: bb as,dn
161:    println(s"'$b'=='$res1' ${b==res1}")
162:    assertEquals(b, res1)
163:  }

Here are the tests:

val b = "bb as,dn"
  test("string trailing"){
    val res0 = "bb as,dn   "
    println(s"'$res0'=='$b' ${res0==b}")
    assertEquals(res0, b)
  }

  test("string leading") {
    val res1 = "   bb as,dn"
    println(s"'$b'=='$res1' ${b==res1}")
    assertEquals(b, res1)
  }

Add failSuite() helper to abort the entire test suite

Currently, you can use fail() to abort a single test case with a custom message. However, the next test case will continue to run unchanged.

It would be nice if there was a failSuite() helper that would abort subsequent test cases as well.

Context: https://contributors.scala-lang.org/t/announcing-munit-a-new-scala-testing-library/4007/5?u=olafurpg

intermittent test failure (TimeoutSuite)

not sure how often it happens, but I did see it once already: https://scala-ci.typesafe.com/job/scala-2.12.x-jdk14-integrate-community-build/699/artifact/logs/munit-build.log

[munit] �[91m==> X �[0m�[91mmunit.TimeoutSuite�[0m.�[91mfast�[0m  �[90m0.025s�[0m java.util.concurrent.TimeoutException: Futures timed out after [10 milliseconds]

Fail on objects that extend munit.Suite

Currently, if the user uses an object instead of class then the test suite silently doesn't run at all. MUnit should report an error and fail the test suite.

Testing for non-compilation wanted

Should help a lot when working with macros or as examples of illegal usage cases instead of documenting them in comments like here

Add Scalafix rule for reporting .only in CI

Just an idea, see scalameta/metals#1946 (comment) for context, but in general .only annotations are useful for local development but should not be committed.

However it's easy to forget to remove them, so a Scalafix linting rule could help and could maybe be shipped with MUnit itself?

Feature Req: Describe section

I know the main philosophy of munit is to keep it lean and simple, so I understand if this feature is not part of that picture--but I'll ask anyway.

There are so many great highlighting features in munit I don't miss my old standby test framework, scalatest a lot. One feature scalatest did have that was nice to have was a describe{} section that allowed tests in a file to be sectioned and intended.

For example:

describe(
    "---------------------------\n:  Scala Primitive Tests  :\n---------------------------"
  ) {
      describe("+++ Positive Tests +++") {
           // several tests here
      }
      describe("+++ Positive Tests +++") {
           // several tests here
      }
}

This would produce a nice sectioned display:

---------------------------
:  Scala Primitive Tests  :
---------------------------
+++ Positive Tests +++
   -- test output
   -- test output
+++ Positive Tests +++
   -- test output
   -- test output

This feature is a nice way to section and format groups of tests. Describe is only an output/formatting thing--it doesn't "do" anything in terms of the tests themselves.

Using the test reporter without using MUnit as test framework

Hi there,

I am trying to use the MUnit sbt plugin as described here in order to generate an HTML test report for my project. I am boldly assuming that I can do so even if I am not using MUnit as the project test framework (my project uses Scalatest instead, which should still generate JUnit compliant XML results).

Sbt changes

I have defined a subproject that depends on another project/module in my build:

lazy val `lenses-domain-core-docs` =
  project
    .in(file("project-x-docs"))
    .dependsOn(`project-x`)
    .settings(
      mdocVariables := Map(
        "VERSION" -> version.value,
      ),
      mdocIn := file("project-x-docs/docs"),
    )
    .enablePlugins(MdocPlugin, MUnitReportPlugin)

When I run sbt 'project-x-docs/mdoc' I can see Mdoc working as expected and compiling scala snippets in my markdown files. However, the HTML report generation fails with a warning:

warning: Unable to generate MUnit HTML test report.

Is there a way to troubleshoot this error, perhaps by making the plugin more verbose?

Google Cloud Storage setup

Notice that, I have exported the GOOGLE_APPLICATION_CREDENTIALS_JSON and GOOGLE_APPLICATION_CREDENTIALS env variable pointing to a service account that has admin access to Google storage on my Gcloud. However, I could not create a bucket named munit-test-reports as I believe these have to be unique across accounts.

Finally, this is the content of the property file generated by MUnit in resource_managed:

❮ cat ./project-x-docs/target/scala-2.12/resource_managed/main/munit.properties
#MUnit properties
#Fri Apr 03 17:59:17 BST 2020
munitBucketName=munit-test-reports

assertNoDiff: java.util.NoSuchElementException: next on empty iterator

==> X tests.ApplicationSuite.trailing-argument  0.002s java.util.NoSuchElementException: next on empty iterator
    at scala.collection.immutable.StringLike$$anon$1.next(StringLike.scala:121)
    at scala.collection.immutable.StringLike$$anon$1.next(StringLike.scala:115)
    at scala.collection.Iterator$$anon$10.next(Iterator.scala:461)
    at munit.internal.difflib.Diff.asStripMargin(Diff.scala:58)
    at munit.internal.difflib.Diff.createReport(Diff.scala:28)
    at munit.internal.difflib.Diffs$.assertNoDiff(Diffs.scala:23)
    at munit.Assertions.$anonfun$assertNoDiff$1(Assertions.scala:53)
    at munit.internal.console.StackTraces$.dropInside(StackTraces.scala:10)
    at munit.Assertions.assertNoDiff(Assertions.scala:53)
    at munit.Assertions.assertNoDiff$(Assertions.scala:47)

Clean up beforeAll,beforeEach,afterAll,afterEach boilerplate in MUnitRunner

There is currently a lot of boilerplate in MUnitRunner for evaluating before/after lifecycle methods. We should be able to incorporate these directly into the bodies of the test methods.

fatal errors thrown in scalacheck mode lose stacks

see this discussion:

https://gitter.im/scalameta/munit?at=5eb9dc2310d5ec031aff33d9

you can repro it with a simple property:

  property("hide my stacks") {
    Prop.forAll { i: Int =>
      if (i > 0) throw new StackOverflowError()
      else Prop(true)
    }
  }

Don't swallow exceptions when they have no stack traces

I've just run into an issue where MUnit wouldn't report an exception thrown when it has no writable stack trace. Libraries such as jsoniter-scala might throw exceptions with no writable stack traces by default to avoid their performance penalty.

You can reproduce this issue with the following snippet.

class SwallowedExceptionSpec extends FunSuite {
  test("i am reported as success but i should be failure") {
    class SwallowedException(msg: String, withStackTrace: Boolean)
      extends RuntimeException(msg, null, false, withStackTrace)
    throw new SwallowedException("i am not reported", false)
  }
}

Changing the previous code to throw new SwallowedException("i am not reported", true) would make munit report the failed test case as expected.

I am using version 0.4.5 and Scala version 2.13.1.

Incorrect class suite name in HTML reports generated with Gradle

HTML test reports generated with Gradle seem to have an incorrect class suite name:

It seems that the class name is the name of the first test case in the suite. In this example it should show "ChainedSuite" instead of "a".

Reproduction

1. Install Gradle 6 (no matter the OS).
2. Download this repository (note the munit branch).
3. Inside the repository folder run:

$ gradle wrapper

4. Generate the reports:

$ ./gradlew clean test

5. Find the reports in repository/scala-only/build/reports/tests/test.

Add to Dotty community build

MUnit stresses a few Dotty features (and we actually hit on and reported a compiler bug when implementing Dotty support) so I think it would make sense to include it in https://github.com/lampepfl/dotty/tree/master/community-build/test/scala/dotty/communitybuild

Cannot get rich filtering capabilities example working

I am really excited about the rich filtering capabilities!

Here's the example on the website that I cannot get working:

import scala.util.Properties
import munit._
object Windows213 extends Tag("Windows213")
class MySuite extends FunSuite {
  // reminder: type Test = GenericTest[Any]
  override def munitNewTest(test: Test): Test = {
    val isIgnored =
      options.tags(Windows213) && !(
        Properties.isWin &&
        Properties.versionNumberString.startsWith("2.13")
      )
    if (isIgnored) test.withBody(() => Ignore)
    else test
  }

  test("windows-213".tag(Windows213)) {
    // Only runs when operating system is Windows and Scala version is 2.13
  }
  test("normal test") {
    // Always runs like a normal test.
  }
}

Changing from options.tags to test.tags solves one of the errors (please confirm this fix is correct).

The if (isIgnored) test.withBody(() => Ignore) line is still erroring out with this message:

[error] /Users/powers/Documents/code/my_apps/munit-example/src/test/scala/com/github/mrpowers/munit/example/RichFiltersSpec.scala:16:40: type mismatch;
[error] found : munit.Tag
[error] required: MySuite.this.TestValue
[error] (which expands to) scala.concurrent.Future[Any]
[error] if (isIgnored) test.withBody(() => Ignore)
[error] ^
[error] one error found
[error] (Test / compileIncremental) Compilation failed

Thanks for building this awesome library!

Add example using MUnit with Maven builds

The documentation currently doesn't explain how to setup MUnit with Maven. Here is an example repo that sets up MUnit with Maven https://gitter.im/scalameta/munit?at=5e9b77505706b414e1daea88

Possibly add native Scala mocking

It might be nice to add a native Scala mocking solution to MUnit. Your scalameta expertise might help ;)

I asked about the future of mocking in Scala and it seems like most folks use mockito-scala.

Looks like mockito-scala might only work with scalatest and specs2 - not sure.

ScalaMock can only mock traits and no-args classes . It looks like they were waiting for scala.meta to be released and then the project lost steam.

I found it easier to write mocks in other languages:

• Python has mocking built into unittest. unittest is built into Python, so all Python projects have easy access to mocking.
• Ruby's rspec mocking is the best mocking syntax / framework I've worked with. Mocking in RSpec didn't require detailed thought like it does for other frameworks.

Let me know what you think!