nomeata / tasty-expected-failure Goto Github PK

View Code? Open in Web Editor NEW

21.0 21.0 9.0 48 KB

Mark test cases as expected-failure

License: MIT License

Haskell 100.00%

tasty-expected-failure's People

Contributors

Stargazers

Watchers

Forkers

jdnavarro expipiplus1 pheaver mpickering unisay sjakobi kquick unkindpartition

tasty-expected-failure's Issues

Omit coverage for `expectFail`ed tests

I did this over in https://github.com/uber/queryparser/blob/master/test/Test/HUnit/Ticket.hs

I think it's as simple as adding the examineTix, clearTix, updateTix, but I wanted to check it'd be desirable before putting together a PR.

`expectFail` and timed-out tests

Can expectFail turn timeout tests to successful too?

Right now, the following code will not succeed:

import Test.Tasty
import Test.Tasty.HUnit
import Test.Tasty.ExpectedFailure (expectFail)
import Control.Concurrent (threadDelay)

main = defaultMain $
        localOption (mkTimeout 1000000) $  -- 1s
          expectFail $ testCase "test1" (threadDelay 2000000) -- 2s

Incompatibility with tasty-1.4

Hi! I'm using tasty-expected-failure to check that my code calls error in some cases. This worked great so far, but after an update to tasty-1.4, the exceptions generated by error are not caught, and instead propagate to tasty which then fails the test.

Minimal, reproducible example

reproducer.cabal

cabal-version:       1.12

name:                reproducer
version:             0.1.0.0
license-file:        LICENSE
author:              Alexander Batischev
maintainer:          [email protected]
build-type:          Simple

library
  exposed-modules:     MyLib
  build-depends:       base >=4.13 && <5
  default-language:    Haskell2010

test-suite reproducer-test
  default-language:    Haskell2010
  type:                exitcode-stdio-1.0
  main-is:             MyLibTest.hs
  build-depends:       base >=4.13 && <5,
                       reproducer,
                       tasty,
                       tasty-hunit ==0.10.0.3,
                       tasty-expected-failure ==0.12.1

MyLib.hs

module MyLib (MyResultType(..), someFunc) where

data MyResultType = MyResultType
  { mrtOne :: String,
    mrtTwo :: String
  }
  deriving (Eq, Show)

someFunc :: String -> MyResultType
someFunc fp =
  MyResultType
    -- Fails.
    { mrtOne = fp,
      mrtTwo = undefined
    }

    -- Curiously, reordering the lines makes it work as expected.
    {-
    { mrtOne = undefined,
      mrtTwo = fp
    }
    -}

MyLibTest.hs

import MyLib
import Test.Tasty (defaultMain, TestTree)
import Test.Tasty.ExpectedFailure (expectFail)
import Test.Tasty.HUnit

main :: IO ()
main = defaultMain myTest

output :: MyResultType
output =
  MyResultType
    { mrtOne = "",
      mrtTwo = ""
    }

myTest :: TestTree
myTest =
  expectFail $
    testCase
      "this is expected to fail"
      (someFunc "Hello, world!" @?= output)

With tasty 1.3.1, I get the expected result — the exception thrown by undefined gets caught by tasty-expected-failure, resulting in "FAIL (expected)" message:

$ cabal test --enable-tests --constrain 'tasty == 1.3.1'
[…]

$ cat dist-newstyle/…/reproducer-0.1.0.0-reproducer-test.log
Test suite reproducer-test: RUNNING...
this is expected to fail: FAIL (expected)
  message threw an exception: Prelude.undefined
  CallStack (from HasCallStack):
    error, called at libraries/base/GHC/Err.hs:80:14 in base:GHC.Err
    undefined, called at ./MyLib.hs:14:16 in main:MyLib

All 1 tests passed (0.00s)
Test suite reproducer-test: PASS

With tasty 1.4.0.1, I get a "FAIL" message instead:

$ cabal test --enable-tests --constrain 'tasty == 1.4.0.1'
[…]
Test suite reproducer-test: RUNNING...
this is expected to fail: FAIL
  Exception: Prelude.undefined
  CallStack (from HasCallStack):
    error, called at libraries/base/GHC/Err.hs:80:14 in base:GHC.Err
    undefined, called at ./MyLib.hs:14:16 in main:MyLib

1 out of 1 tests failed (0.00s)
Test suite reproducer-test: FAIL

I narrowed it down to UnkindPartition/tasty@fcfccc8. If I manually use that commit's parent, everything works as expected (the first listing above):

cabal.project

packages: .

source-repository-package
    type: git
    location: https://github.com/feuerbach/tasty
    tag: 96795203335a54d39e3587c890c2879a3c1a0150
    subdir: core

If I specify tag: fcfccc8ea991583d649f5129b83660d6cee38d8b instead, I get the unexpected behaviour (second listing above).

Looking at the offending commit, I don't see how it could cause this regression. If anything, it should force exceptions to be thrown sooner, enabling tasty-expected-failure to catch more of them — but the situation is exactly the opposite now.

tasty-1.4 have been out for a month now, tasty-expected-result got downloaded 245 times in the last 30 days, and no-one reported this yet, so I'm assuming my code is broken somehow?

I'm at my wits' end here. Does anyone have any idea as to what's going on and how to fix it?

expected-fail-hh-tests failed in 0.12.1

Test suite expected-fail-hh-tests: RUNNING...
Expected Hedgehog Failures
  good:                                OK
      ✓ good passed 100 tests.
  rarely good:                         FAIL (expected)
      ✗ rarely good failed at tests/TestExpectedFailsHH.hs:22:7
        after 25 tests and 7 shrinks.

           ┏━━ tests/TestExpectedFailsHH.hs ━━━
        16 ┃ main = defaultMain $
        17 ┃   localOption (mkTimeout 1000000) $  -- 1s
        18 ┃   testGroup "Expected Hedgehog Failures" $
        19 ┃   [ testProperty "good" $ property $ success
        20 ┃   , expectFail $ testProperty "rarely good" $ property $ do
        21 ┃       xs <- forAll $ Gen.list (Range.linear 0 10) Gen.alpha
           ┃       │ "ab"
        22 ┃       reverse xs === xs
           ┃       ^^^^^^^^^^^^^^^^^
           ┃       │ ━━━ Failed (- lhs) (+ rhs) ━━━
           ┃       │ - "ba"
           ┃       │ + "ab"
        23 ┃
        24 ┃   -- n.b. uncomment this to observe the results of a test that was
        25 ┃   -- expected to fail but actually passes.
        26 ┃   -- , expectFail $ testProperty "surprisingly good" $ property $ success
        27 ┃
        28 ┃   , expectFail $ testProperty "giving up" $ property $ discard
        29 ┃
        30 ┃   , expectFail $ expectFail $ testProperty "the failure of a failure is my good" $
        31 ┃     property $ success
        32 ┃
        33 ┃   , expectFail $ testProperty "throws failure" $
        34 ┃     property $ fail "bad"
        35 ┃
        36 ┃   , expectFail $ testProperty "too slow" $
        37 ┃     property $ do
        38 ┃       liftIO $ threadDelay 2000000
        39 ┃       success
        40 ┃   ]

        This failure can be reproduced by running:
        > recheck (Size 24) (Seed 10314491015417403926 10569457720475682929) rarely good

    Use '--hedgehog-replay "Size 24 Seed 10314491015417403926 10569457720475682929"' to reproduce.
     (expected failure)
  giving up:                           FAIL (expected)
      ⚐ giving up gave up after 100 discards, passed 0 tests. (expected failure)
  the failure of a failure is my good: OK (unexpected) (expected)
      ✓ the failure of a failure is my good passed 100 tests. (unexpected success) (expected failure)
  throws failure:                      FAIL (expected)
      ✗ throws failure failed
        after 1 test.

        bad

        This failure can be reproduced by running:
        > recheck (Size 0) (Seed 13804550014081903080 14324880539018441715) throws failure

    Use '--hedgehog-replay "Size 0 Seed 13804550014081903080 14324880539018441715"' to reproduce.
     (expected failure)
  too slow:                            TIMEOUT (1.00s)
    Timed out after 1s

1 out of 6 tests failed (1.01s)
Test suite expected-fail-hh-tests: FAIL

expectFail inconsistent with exceptions

Hello,

I'm having some trouble with expectFail when the cause of the failure is an (expected) exception.
Specifically, applying it to a testProperty works fine, but it doesn't if I apply it to a testCase.

In this repository there is a working example of what I mean: expectFail-test

At the moment my hacky solution is to use wrapTest to try to catch all exceptions and then handle them by their printed representation, which feels very fragile:

expectException :: String -> TestTree -> TestTree
expectException s = expectExceptions [s]

expectExceptions :: [String] -> TestTree -> TestTree
expectExceptions prefixes = wrapTest (handleJust selector handler)
  where
    selector :: SomeException -> Maybe SomeException
    selector e
      | not . null . prefixesMatching $ e = Just e
      | otherwise = Nothing
    handler e =
      pure
        Result
          { resultOutcome = Success,
            resultDescription = "Exception matches: " <> (intercalate " OR " . map quoted) (prefixesMatching e),
            resultShortDescription = "FAIL (expected)",
            resultTime = 0
          }
    prefixesMatching e = filter (`isPrefixOf` show e) prefixes
    quoted s = "\"" <> s <> "\""

Indeed, it only solves the cases where expectFail does not work, and it does not work where expectFail operates as intended.

My understanding so far is that testProperty somehow catches all exceptions and returns a Failure result, while testCase does no such thing and raises an Exception.
Is there a way to have a (robust) wrapper of TestTree so it can catch Exceptions consistently?

Exceptions aren't counted as expected failures with tasty-golden

module Main where

import Test.Tasty
import Test.Tasty.ExpectedFailure
import Test.Tasty.Golden

main :: IO ()
main = defaultMain $ expectFail $
  goldenVsStringDiff "hello" (\ref new -> ["diff", "-u", ref, new]) "hello.out" $
    return $ error "This is an error."
$ cabal run
Preprocessing executable 'tasty-debug' for tasty-debug-0.1.0.0...
Running tasty-debug...
hello: FAIL
  Exception: This is an error.
  CallStack (from HasCallStack):
    error, called at Main.hs:10:14 in main:Main

1 out of 1 tests failed (0.00s)

This is with tasty 0.11.2.1, tasty-golden 2.3.1.1 and tasty-expected-failure 0.11.0.4.

Test failure in 0.12

I am not sure what is the cause, though. All packages are at latest versions.

Running 2 test suites...
Test suite expected-fail-tests: RUNNING...
Expected Failures
  clearly good:            OK
  clearly bad:             FAIL (expected)
    tests/TestExpectedFails.hs:17:
    expected: 2
     but got: 3 (expected failure)
  two wrongs make a right: OK (unexpected) (expected)
     (unexpected success) (expected failure)
  throws failure:          FAIL (expected)
    Exception: user error (bad) (expected failure)
expected-fail-tests: Test/Tasty/ExpectedFailure.hs:(48,13)-(52,20): Missing field in record construction resultDetailsPrinter

Test suite expected-fail-tests: FAIL
Test suite logged to:
dist/test/tasty-expected-failure-0.12-expected-fail-tests.log
Test suite expected-fail-hh-tests: RUNNING...
Test suite expected-fail-hh-tests: PASS
Test suite logged to:
dist/test/tasty-expected-failure-0.12-expected-fail-hh-tests.log
1 of 2 test suites (1 of 2 test cases) passed.

base 4.11

tasty-expected-failure should work OK on GHC 8.4 (at least, none of my builds with --allow-newer broke because of it), but it's currently held back by the upper bound on base.

setup/teardown is executed for ignored tests?

Hello, it appears that ignoreTest t executes any setup/teardown (via withResource). Is this intentional?

I don't think that's desired behavior for us. The setup and teardown are part of the test that we need to skip (e.g. in one case, we want to use a flag to disable a test whose withResource call would blow up).

ignore at runtime

I have a test case which may not be able to test anything, but that can't be determined until the IO action of the test case is run.

A simple example of this is a test that relies on some external resource that it may or may not be able to allocate.. say a certian amount of free disk space.

It would be useful if tasty-expected-failure could support this.

I have not tried to implement it, but it looks like it just needs a function to generate a Result like the one ignoreTest passes to wrapTest.

Add `expectAnyFail`

allowFail in #1 will still succeed if all tests under a TestTree pass. However, it'd useful to have another test modifier, i.e. expectAnyFail, that reports a failure when all tests succeed but passes when at least one test fails.

Exception throwing tests still fail

The test error "fail" wrapped in expectFail will still be marked as failing.

nomeata / tasty-expected-failure Goto Github PK

tasty-expected-failure's People

Contributors

Stargazers

Watchers

Forkers

tasty-expected-failure's Issues

Recommend Projects

Recommend Topics

Recommend Org

Jobs