failed assertions in future context show as passing about scalatest-junit-runner

[giurim]

Hi Gabriel, Thank you for the report and the examination. I am away but I will look into this next week. Intellij might call a different "entry point" hence the difference in reporting. Gyuri

…

On Tue, Aug 3, 2021, 19:09 Gabriel Jones ***@***.***> wrote: After closer examination, I realize that assertions inside of a future.failed.foreach is a definite code smell. Somehow Intellij is able to see the TestFailedException even though it was thrown in some other thread and mark the test failed anyway. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#60 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADST4FLSB2ZLP2B3HRLQHSTT3APEFANCNFSM5BPGN3CA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email> .

[giurim]

Hi Gabriel,

I just checked what is going on with this issue. As far as I understand it, this problem falls outside of the scope of the test runner.

The foreach function catches all exceptions and reports them to the execution context. In the javadoc it is described as

Since this method executes asynchronously and does not produce a return value, any non-fatal exceptions thrown will be reported to the ExecutionContext.

The default handler prints a stack-trace but otherwise does nothing else. The type of the ExecutionContext used determines how this error will be handled. Also because of the async execution of the lambda of the foreach, the test itself already finishes with success when the assertions are running.

I think using foreach is not the best approach because of how the error is handled. I am not sure if this is the best solution, but using map instead of foreach and waiting on the result of the assertions broke the test when the assertion did break:

test("should fail future") {
    val future = Future {
      1
    }

    Await.result(
      future.map({ r =>
        r should ===(2)
      }),
      Duration.Inf)
  }

Let me know if you find anything else to handle these assertions.
Best,
Gyuri

[gabrieljones]

After closer examination, I realize that assertions inside of a future.foreach is a definite code smell. Somehow Intellij is able to see the TestFailedException even though it was thrown in some other thread and mark the test failed anyway.