skogsbaer / htf Goto Github PK

Running htfMain mytests under Windows in GHCi causes an exception:

*** Exception: .HTF/<interactive>.history: openBinaryFile: invalid argument (Invalid argument)

The cause is quite simple. From Test/Framework/CmdlineOptions.hs:339:

  getHistoryFile =
      case opts_historyFile opts of
        Just fp -> return fp
        Nothing ->
            do x <- getProgName
               createDirectoryIfMissing False ".HTF"
               return (".HTF/" ++ x ++ ".history")

The issue is that under GHCi, getProgName returns the string "<interactive>", but < and > are invalid characters for a filename under Windows. I would suggest a simple filter to remove the set of invalid filename characters under Windows would be the best solution (see https://msdn.microsoft.com/en-gb/library/windows/desktop/aa365247(v=vs.85).aspx for a list).

In the HTF tutorial there is a small bug, this sample code does not work:

prop_reverseReplay =
  withQCArgs (a -> a { replay = read "Just (1060394807 2147483396,2)" })
  prop_reverse

A backslash is missing in line to in front of the first a, should be

prop_reverseReplay =
  withQCArgs (\a -> a { replay = read "Just (1060394807 2147483396,2)" })
  prop_reverse

A nice set of additional assertion combinators would be something like:

assertEqualM :: (Eq a, Show a) => a -> IO a -> IO () 

Some QuickCheck stuff doesn't work in HTF. For example using the ==> operator:

prop_expandWindow2 s n = n < 10 ==> expandWindowPre1 n w == expandWindowPost1 w
…
    • Couldn't match expected type ‘Bool’ with actual type ‘Property’

This is likely true for forAll also, since those are the two combinators that are exceptions to the "properties must be Bool" rule in QuickCheck:

The result type of a property should be Bool, unless the property is defined using other combinators below.
http://www.cse.chalmers.se/~rjmh/QuickCheck/manual.html

I'm not super-familiar with the internals ot HTF or QuickCheck but might this be resolved by adding an instance of the type class Testable for the class Property?

The following stackoverflow questions summarizes the issue very well: http://stackoverflow.com/questions/14044356/haskell-test-framework-htf-and-test-definitions-in-multiple-modules

Proposed solution:

• htf_thisModulesTest should be of type [TestSuite] (and not of type TestSuite as it is now)
• If a module imports tests via the HTF_TESTS pragma, then htf_importedTests should be appended to htf_thisModulesTests

HTF-0.10.0.7 build seems broken with haskell-src-exts-1.14.0

 48 [16 of 17] Compiling Test.Framework   ( Test/Framework.hs, dist/build/Test/F    ramework.p_o )
 49 [17 of 17] Compiling Test.Framework.Tutorial ( Test/Framework/Tutorial.hs, d    ist/build/Test/Framework/Tutorial.p_o )
 50 In-place registering HTF-0.10.0.7...
 51 Preprocessing executable 'htfpp' for HTF-0.10.0.7...
 52 [1 of 6] Compiling Test.Framework.Utils ( Test/Framework/Utils.hs, dist/buil    d/htfpp/htfpp-tmp/Test/Framework/Utils.o )
 53 [2 of 6] Compiling Test.Framework.Location ( Test/Framework/Location.hs, dis    t/build/htfpp/htfpp-tmp/Test/Framework/Location.o )
 54 [3 of 6] Compiling Test.Framework.HaskellParser ( Test/Framework/HaskellPars    er.hs, dist/build/htfpp/htfpp-tmp/Test/Framework/HaskellParser.o )
 55 
 56 Test/Framework/HaskellParser.hs:112:41:
 57     Couldn't match expected type `Ext.Extension'
 58                 with actual type `Ext.KnownExtension'
 59     In the expression: Ext.BangPatterns
 60     In the second argument of `(++)', namely
 61       `[Ext.BangPatterns, Ext.TemplateHaskell]'
 62     In the `Parser.extensions' field of a record
 63 
 64 Test/Framework/HaskellParser.hs:112:59:
 65     Couldn't match expected type `Ext.Extension'
 66                 with actual type `Ext.KnownExtension'
 67     In the expression: Ext.TemplateHaskell
 68     In the second argument of `(++)', namely
 69       `[Ext.BangPatterns, Ext.TemplateHaskell]'
 70     In the `Parser.extensions' field of a record

If a source file contains non-ASCII characters, even in comments, then any tests that come after those characters are not picked up into htf_thisModulesTests.

Example file:

{-# OPTIONS_GHC -F -pgmF htfpp #-}

module Main where

import Test.Framework

test_first = do
    -- 世界不是英文的
    assertEqual True True

test_second = do
    assertEqual True False

main = htfMain htf_thisModulesTests

Output when run:

[TEST] Main:first (htf-unicode-bug.hs:7)
+++ OK (0ms)

* Tests:     1
* Passed:    1
* Pending:   0
* Failures:  0
* Errors:    0
* Timed out: 0
* Filtered:  0

Total execution time: 0ms

Output of htfpp on the file:

#line 1 "<stdin>"
{-# OPTIONS_GHC -F -pgmF htfpp #-}

module Main where

import Test.Framework

test_first = do
    -- 世界不是英文的
    (assertEqual_ (makeLoc "<stdin>" 9)) True True

test_second = do
    (assertEqual_ (makeLoc "<stdin>" 12)) True False

main = htfMain htf_Main_thisModulesTests


htf_Main_thisModulesTests :: TestSuite
htf_Main_thisModulesTests = makeTestSuite "Main" [
{-# LINE 7 "<stdin>" #-}
    makeUnitTest "first" (makeLoc "<stdin>" 7) test_first
  ]

Note how test_second is missing from htf_Main_thisModulesTests.

It should be possible to define benchmarks in HTF. Benchmark should use criterion for measurements.

Possibility 1:

bench_fact = nf fact 100

This simply runs the benchmark when explicitly requested but not when executing the whole test suite.

Possibility 2:

bench_fact = withReference someReferenceBenchmark 3.4 $ nf fact 100

This checks that running nf fact 100 is at most 3.4 times as slow as `someReferenceBenchmark. This benchmark is run when the whole test suite is run.

Benchmark should not be executed concurrently to other tests.

In stackage nightly:

Building library for HTF-0.13.2.2..
[ 1 of 23] Compiling Test.Framework.Colors ( Test/Framework/Colors.hs, dist/build/Test/Framework/Colors.o )
[ 2 of 23] Compiling Test.Framework.Diff ( Test/Framework/Diff.hs, dist/build/Test/Framework/Diff.o )
[ 3 of 23] Compiling Test.Framework.Location ( Test/Framework/Location.hs, dist/build/Test/Framework/Location.o )
[ 4 of 23] Compiling Test.Framework.Preprocessor ( Test/Framework/Preprocessor.hs, dist/build/Test/Framework/Preprocessor.o )

Test/Framework/Preprocessor.hs:46:1: warning: [-Wdodgy-imports]
    Module ‘Test.HUnit’ does not export ‘Location’
   |
46 | import Test.HUnit hiding (State, Location)
   | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[ 5 of 23] Compiling Test.Framework.Pretty ( Test/Framework/Pretty.hs, dist/build/Test/Framework/Pretty.o )

Test/Framework/Pretty.hs:32:18: error:
    Ambiguous occurrence ‘<>’
    It could refer to either ‘Prelude.<>’,
                             imported from ‘Prelude’ at Test/Framework/Pretty.hs:13:8-28
                             (and originally defined in ‘GHC.Base’)
                          or ‘Text.PrettyPrint.<>’,
                             imported from ‘Text.PrettyPrint’ at Test/Framework/Pretty.hs:22:1-23
                             (and originally defined in ‘Text.PrettyPrint.HughesPJ’)
   |
32 |         char '[' <> vcat (punctuate comma (map pretty l)) <> char ']'
   |                  ^^

Test/Framework/Pretty.hs:32:59: error:
    Ambiguous occurrence ‘<>’
    It could refer to either ‘Prelude.<>’,
                             imported from ‘Prelude’ at Test/Framework/Pretty.hs:13:8-28
                             (and originally defined in ‘GHC.Base’)
                          or ‘Text.PrettyPrint.<>’,
                             imported from ‘Text.PrettyPrint’ at Test/Framework/Pretty.hs:22:1-23
                             (and originally defined in ‘Text.PrettyPrint.HughesPJ’)
   |
32 |         char '[' <> vcat (punctuate comma (map pretty l)) <> char ']'
   |                                                           ^^

Hi. I'm trying to use HTF with EclipseFP and I'm running into trouble. I've already asked on the EclipseFP discussion forum but so far there is no solution, and it was suggested to ask here in case anyone had an idea.

Basically EclipseFP fails to use HTF, Apparently it tries to invoke htfpp through buildwrapper and fails with this message:

buildwrapper: buildwrapper: could not execute: htfpp

However, htfpp (as well as buildwrapper) is on ~/.cabal/bin which is on my path.

Furthermore, htfpp works fine when invoked on the command-line.

This is on a fresh installation of the haskell-platform (ghc-7.4.2, cabal 1.14.0, HTF 0.10.0.7, buildwrapper 0.6.4) with EclipseFP 2.4.2 on Eclipse 4.2.1 on Ubuntu 12.10.

Any ideas on what could be the problem?

Thanks

I.e., HTF counts the following as two tests in the report and it runs it twice:

prop_updatesProduceTheSameEffectAsInHashMap :: Update.Update Word8 Char -> Bool
prop_updatesProduceTheSameEffectAsInHashMap update = ...

HTF 0.11.3.4

Preprocessing library HTF-0.11.3.1...

Test/Framework.hs:46:8:
    Could not find module `Test.Framework.AssertM'
    Perhaps you meant
      Test.Framework.Seed (needs flag -package test-framework-0.8.0.3)
    Use -v to see a list of the files searched for.
Failed to install HTF-0.11.3.1
cabal: Error: some packages failed to install:
HTF-0.11.3.1 failed during the building phase. The exception was:
ExitFailure 1

Reproduced on both GHC 7.4 and 7.6.

Running:

$ cabal install HTF --constraint 'QuickCheck == 2.6'

Ultimately ends with

[21 of 23] Compiling Test.Framework.QuickCheckWrapper ( Test/Framework/QuickCheckWrapper.hs, dist/build/Test/Framework/QuickCheckWrapper.o )

Test/Framework/QuickCheckWrapper.hs:203:14:
    Not in scope: ‘morallyDubiousIOProperty’
Failed to install HTF-0.12.2.0
cabal: Error: some packages failed to install:
HTF-0.12.2.0 failed during the building phase. The exception was:
ExitFailure 1

Performed with GHC 7.8.3.

I imply something in the spirit of the following, but with a proper location and htfpp support:

assertElem item list =
  assertBoolVerbose 
    ("Expected item '" ++ item ++ "' to be in " ++ show list)
    (item `elem` list)

I personally find myself very often in the need of this.

When I use lens's ^. in HTF tests like that:

a ^. b . c

I get the following error from htfpp:

htfpp warning: Parsing of test-suites/main/Test/RadioX/ProcessedCatalogue/Model/ProcessedCatalogue.hs failed at line 0: Ambiguous infix expression

If however I introduce some braces, the parsing goes fine:

a ^. (b . c)

Test/Framework/HaskellParser.hs:112:41:
Couldn't match expected type Ext.Extension' with actual typeExt.KnownExtension'
In the expression: Ext.BangPatterns
In the second argument of (++)', namely [Ext.BangPatterns, Ext.TemplateHaskell]'
In the `Parser.extensions' field of a record

Test/Framework/HaskellParser.hs:112:59:
Couldn't match expected type Ext.Extension' with actual typeExt.KnownExtension'
In the expression: Ext.TemplateHaskell
In the second argument of (++)', namely [Ext.BangPatterns, Ext.TemplateHaskell]'
In the `Parser.extensions' field of a record

aeson-pretty changed Config multiple times, so tests/TestHTF.hs fails to build with 0.8.2 and 0.8.5

Tests should be executed in parallel. Take care that output from tests is still readable! Also, certain tests don't want to be executed in parallel (e.g. benchmarks).

Hello,

When I do cabal install HTF I see this output:

In-place registering xmlgen-0.6.2.1...
Installing library in /root/.cabal/lib/xmlgen-0.6.2.1/ghc-7.6.3
Registering xmlgen-0.6.2.1...
Installed xmlgen-0.6.2.1
Downloading HTF-0.11.3.4...
Configuring HTF-0.11.3.4...
Building HTF-0.11.3.4...
Preprocessing library HTF-0.11.3.4...
ghc: could not execute: cpphs
Failed to install HTF-0.11.3.4
cabal: Error: some packages failed to install:
HTF-0.11.3.4 failed during the building phase. The exception was:
ExitFailure 1

cpphs is installed see this output:
cabal install cpphs
Resolving dependencies...
All the requested packages are already installed:
cpphs-1.18.4
Use --reinstall if you want to reinstall anyway.

Roelof

Support for running tests in a random order.

Currently, HTF is a big, monolithic package. It would be nice to split HTF into several smaller packages. This has the following benefits:

• extensibility: e.g. support for other test methods such as smallcheck
• possibly fewer dependencies
• choices for the user: for example, the user can decide whether to have source code locations in assertion failures, and how these source code locations are realized (template haskell, htfpp, or something different)

I could imagine the following packages:

HTF-core

The core of HTF. Defines how tests are organized and executed. This could include the following modules:

• Test.Framework.AssertM
• Test.Framework.History
• Test.Framework.JsonOutput
• Test.Framework.Location
• Test.Framework.CmdlineOptions
• Test.Framework.Colors
• Test.Framework.Pretty
• Test.Framework.PrettyHaskell
• Test.Framework.TestInterface
• Test.Framework.TestManager
• Test.Framework.TestReporter
• Test.Framework.TestTypes
• Test.Framework.XmlOutput
• Test.Framework.ThreadPool

We could split HTF-core even in even smaller parts by abstracting over how test results are reported. I think we should post this split.

Things to consider:

• It should be possible to write and execute simple tests with HTF-core.

HTF-UnitTests

HTF adapter for HUnit. Includes the current module

• Test.Framework.HUnitWrapper

HTF-QuickCheck

HTF adapter for QuickCheck. Includes the current module

• Test.Framework.QuickCheckWrapper

HTF-ShellTests

Shell-based tests with HTF. Includes the current module

• Test.Framework.BlackBoxTest

(I suggest renaming "black box tests" to "shell tests")

HTF-preprocessor

The htfpp preprocessor. Includes the current module

• Test.Framework.Preprocessor

htfpp needs a configuration file where we define the prefix to look for and the expansion code for a given prefix. This is necessary to be able to extend htfpp to new test providers (e.g. smallcheck). There should be a sensible default configuration file for all know test providers.

HTF

The HTF umbrella package, defining a sensible default set of package dependencies. Its functionality is the same as the current HTF package. Also includes the tutorial:

• Test.Framework.Tutorial

I am a bit confused by the description here:

Suppose that one of the .num files is bbt-dir/should-pass/x.num. Running the corresponding Test invokes dist/build/sample/sample (the program under test) with bbt-dir/should-pass/x.num as input file. If bbt-dir/should-pass/x.num existed, its content would be used as stdin. The tests succeeds if the exit code of the program is zero and the output on stdout and stderr matches the contents of bbt-dir/should-pass/x.out and bbt-dir/should-pass/x.err, respectively.

It assumes that bbt-dir/should-pass/x.num exists to run the test and then says that if it exists, it is used as stdin. Does that mean that it is not possible to run a black box test without giving it some stdin content?

And is the file given as an argument or on stdin?
i.e.
dist/build/sample/sample bbt-dir/should-pass/x.num
or
dist/build/sample/sample < bbt-dir/should-pass/x.num

Due to #10 on every compilation I am getting the following messages:

Preprocessing test suite 'remotion-htf-test-suite' for remotion-0.1.0...
htfpp warning: Parsing of src/HTFTestSuite/Setup1.hs failed at line 25: Parse error: case
Falling back to poor man's parser. This parser may return incomplete results. The result returned was: 
Prefix: Test.Framework.
Module name: HTFTestSuite.Setup1
Definitions: []
HTF imports: []
htfpp warning: Parsing of src/HTFTestSuite/Setup1.hs failed at line 25: Parse error: case
Falling back to poor man's parser. This parser may return incomplete results. The result returned was: 
Prefix: Test.Framework.
Module name: HTFTestSuite.Setup1
Definitions: []
HTF imports: []
Running 1 test suites...

This is very annoying. As a user I really am not at all interested, which parsing mode HTF is in.

I suggest to set it up so that no warnings are shown if the fallback parser succeeds.

I have some tests that each need a database pool, but I'd rather not create an individual database pool within each test for performance reasons.

I'd like to write my tests like prop_foo :: a -> Property, and have some way to feed the result of my pool creation function mkPool :: IO a into each test.

Does HTF provide a way to do that and still use its TH test discovery? Even if one has to do discovery manually, is there a way to at least keep the same test settings and pretty results printing code as the rest of one's HTF tests?

1. Open "http://www.cs.chalmers.se/~rjmh/QuickCheck/"
2. You'll get "You don't have permission to access /~rjmh/QuickCheck/ on this server."

The haskell-src-exts dependency takes very long to compile and I think you can still reasonably well discover tests without fully parsing every source file. Maybe copy the idea of http://hspec.github.io/ ?

i use HTF with the cabal test-suite - very convenient.in the package are a number of tests and i would like to use the --quiet command line flag. how to pass this to the test-suite?
thank you for a great tool!
andrew

On the homepage
(http://www.stefanwehr.de/software/) the "blog article" hyperlink is out
of date, and 404s. You could update the link to
http://factisresearch.blogspot.co.uk/2010/03/htf-test-framework-for-haskell.html,
but maybe it'd be best to point it to
http://hackage.haskell.org/packages/archive/HTF/latest/doc/html/Test-Framework-Tutorial.html.

Noticed when building Stackage, but it seems to fail even with a normal, unsandbox build. Running cabal install --enable-tests HTF results in:

Running 1 test suites...
Test suite TestHTF: RUNNING...
Test1.hs:7:13:
Test2.hs:7:13:
Compile error for Test3.hs did not occur in line 7, exit code of grep: 1

<no location info>: can't find file: Test3.hs
Test suite TestHTF: FAIL
Test suite logged to: dist/test/HTF-0.10.0.3-TestHTF.log
0 of 1 test suites (0 of 1 test cases) passed.
Failed to install HTF-0.10.0.3
cabal: Error: some packages failed to install:
HTF-0.10.0.3 failed during the tests phase. The exception was:
ExitFailure 1

This appears to be a case of missing source files in the sdist, but I'm not certain.

Let me know if I can provide any more information.

Consider this example:

test_curryUncurry :: IO ()
test_curryUncurry =
    do assertEqual "12" (fun (1 :&: 2 :&: empty))
       assertEqual "12" (HV.curry fun 1 2)
       assertEqual "12" (HV.uncurry (HV.curry fun) (1 :&: 2 :&: empty))
    where
      fun :: HVect '[Int, Int] -> String
      fun (a :&: b  :&: HNil) = show a ++ show b
      fun _ = "OOPS!"

test_length :: IO ()
test_length =
    do assertEqual 0 (sNatToInt $ HV.length empty)
       assertEqual 2 (sNatToInt $ HV.length ("foo" :&: "bar" :&: empty))

When compiling, I get a warning, that test_length is defined but never used. Probably the preprocessor aborts after HVect '[Int, Int]. Removing that everything works as expected.

I tried to package HTF for NixOS, but noticed that the package's test suite doesn't succeed: http://hydra.cryp.to/build/90103/nixlog/2/raw. Is that a known issue?

There should be a commandline option to repeat all tests specified several times. Example:

./MyTests --repeat=20 foobar

This command would run all tests 20 times that match the name pattern foobar.

HTF declares containers (>=0.4.2.1) as dependency but the latest version make use of Data.Map.Strict, which is introduced in 0.5.0.0.

This causes a problem with ghc 7.4 because cabal selects 0.4.2.1 and tries to build HTF 0.12.0.0 with it.

I'm trying to build some tests for some error conditions. The code that I am testing reports runtime errors using 'System.Exit.exitFailure', which is internally equivalent to "throwIO $ ExitFailure 1". I am unable to catch 'exitFailure' using HTF, but I was hoping I would be able to.

I'm a bit out my depth here, so this might not be a "real" bug. I might be making a rookie mistake. But even then, perhaps this could be addressed with a small documentation comment?

Minimal test case follows. Test case results are the same with (-O, -O1, -O2) and without (-O0) optimization:

{-# OPTIONS_GHC -F -pgmF htfpp #-}
module Main (main) where

import qualified Control.Exception as E
import Test.Framework

main = htfMain testDoesItThrow

-- I want this test to pass.
testDoesItThrow = assertThrowsSome throwSomeException


throwSomeException :: IO ()
-- With this implementation, the test passes; the exception is caught by
-- assertThrowsSome:
-- throwSomeException = E.throw E.Overflow

-- With this implementation, the test is a Failure, saying "no exception
-- was thrown":
throwSomeException = E.throwIO E.Overflow

Discovered during a Stackage build, it's the common Prelude.catch issue:

Component build order: library, executable 'htfpp'
creating dist/build
creating dist/build/autogen
Building HTF-0.11.2...
Preprocessing library HTF-0.11.2...
Building library...
'/opt/ghc-7.4.2/bin/ghc' '--info'
'/opt/ghc-7.4.2/bin/ghc' '--info'
creating dist/build
/opt/ghc-7.4.2/bin/ghc --make -fbuilding-cabal-package -O -outputdir dist/build -odir dist/build -hidir dist/build -stubdir dist/build -i -idist/build -i. -idist/build/autogen -Idist/build/autogen -Idist/build -optP-include -optPdist/build/autogen/cabal_macros.h -package-name HTF-0.11.2 -hide-all-packages -no-user-package-conf -package-conf /var/lib/jenkins/workspace/Stackage with GHC 7.4.2 no HP/stackage/sandbox/package-db -package-conf dist/package.conf.inplace -package-id Diff-0.3.0-35ceea703174d47626e6549f22b2da02 -package-id HUnit-1.2.5.2-246d97748b0329785b0bade80cbd23b4 -package-id QuickCheck-2.5.1.1-8b550a1d042990b5baac58a3e00c4c61 -package-id aeson-0.6.2.1-ff0934437b51ba29d55904b912f2abc1 -package-id array-0.4.0.0-0b6c5ca7e879a14d110ca4c001dd9297 -package-id base-4.5.1.0-66f22db3dfcd87541c9c7e50e7095d26 -package-id bytestring-0.9.2.1-503e91bb155301fdb1956cb5c26ce6e9 -package-id containers-0.4.2.1-75f143aa39a3e77a1ce2300025bdd8ce -package-id directory-1.1.0.2-90bfc47dbd5ac1313aec9ce76cc22338 -package-id haskell-src-exts-1.14.0.1-fc3b421879fe65ce65174a95c15c086f -package-id lifted-base-0.2.1.1-fc4118545856fb20b245f64f2ac0c64c -package-id monad-control-0.3.2.2-fd728c3b1a90294aa5e40b4508a9adf7 -package-id mtl-2.1.2-02e701f9b1590ee88a0b5b0bd5d93a29 -package-id old-time-1.1.0.0-703543375fafbcbe67bc51a758e84f10 -package-id pretty-1.1.1.0-91ed62f0481a81d292d550eec35ee75b -package-id process-1.1.0.1-608c248c1528f46eb960a08cead77291 -package-id random-1.0.1.1-c7787dee61cd418f6edd479355fe381c -package-id regex-compat-0.95.1-cc3d44a316fdd7f574afda5ba3da1be4 -package-id text-0.11.3.1-41f05eb413a8fa91f09d91d3051e4ae4 -package-id unix-2.5.1.1-5b2e7ecb1dee2b50fe019fbe38b88a3e -package-id xmlgen-0.6.2.1-67d4b2ccec1a299aa7b5b640f978cf1b -XHaskell2010 Test.Framework Test.Framework.HUnitWrapper Test.Framework.TestManager Test.Framework.TestTypes Test.Framework.TestReporter Test.Framework.CmdlineOptions Test.Framework.QuickCheckWrapper Test.Framework.BlackBoxTest Test.Framework.Location Test.Framework.Tutorial Test.Framework.Pretty Test.Framework.JsonOutput Test.Framework.XmlOutput Test.Framework.ThreadPool Test.Framework.TestManagerInternal Test.Framework.Utils Test.Framework.Colors Test.Framework.Diff Test.Framework.Process -W -fwarn-unused-imports -fwarn-unused-binds -fwarn-unused-matches -fwarn-unused-do-bind -fwarn-wrong-do-bind
[ 1 of 19] Compiling Test.Framework.Tutorial ( Test/Framework/Tutorial.hs, dist/build/Test/Framework/Tutorial.o )
[ 2 of 19] Compiling Test.Framework.Process ( Test/Framework/Process.hs, dist/build/Test/Framework/Process.o )
[ 3 of 19] Compiling Test.Framework.ThreadPool ( Test/Framework/ThreadPool.hs, dist/build/Test/Framework/ThreadPool.o )
[ 4 of 19] Compiling Test.Framework.Utils ( Test/Framework/Utils.hs, dist/build/Test/Framework/Utils.o )
[ 5 of 19] Compiling Test.Framework.Pretty ( Test/Framework/Pretty.hs, dist/build/Test/Framework/Pretty.o )
[ 6 of 19] Compiling Test.Framework.Colors ( Test/Framework/Colors.hs, dist/build/Test/Framework/Colors.o )
[ 7 of 19] Compiling Test.Framework.Diff ( Test/Framework/Diff.hs, dist/build/Test/Framework/Diff.o )
[ 8 of 19] Compiling Test.Framework.Location ( Test/Framework/Location.hs, dist/build/Test/Framework/Location.o )
[ 9 of 19] Compiling Test.Framework.TestTypes ( Test/Framework/TestTypes.hs, dist/build/Test/Framework/TestTypes.o )
[10 of 19] Compiling Test.Framework.TestManagerInternal ( Test/Framework/TestManagerInternal.hs, dist/build/Test/Framework/TestManagerInternal.o )
[11 of 19] Compiling Test.Framework.XmlOutput ( Test/Framework/XmlOutput.hs, dist/build/Test/Framework/XmlOutput.o )
[12 of 19] Compiling Test.Framework.JsonOutput ( Test/Framework/JsonOutput.hs, dist/build/Test/Framework/JsonOutput.o )
[13 of 19] Compiling Test.Framework.TestReporter ( Test/Framework/TestReporter.hs, dist/build/Test/Framework/TestReporter.o )
[14 of 19] Compiling Test.Framework.CmdlineOptions ( Test/Framework/CmdlineOptions.hs, dist/build/Test/Framework/CmdlineOptions.o )
[15 of 19] Compiling Test.Framework.TestManager ( Test/Framework/TestManager.hs, dist/build/Test/Framework/TestManager.o )
[16 of 19] Compiling Test.Framework.BlackBoxTest ( Test/Framework/BlackBoxTest.hs, dist/build/Test/Framework/BlackBoxTest.o )
[17 of 19] Compiling Test.Framework.QuickCheckWrapper ( Test/Framework/QuickCheckWrapper.hs, dist/build/Test/Framework/QuickCheckWrapper.o )
[18 of 19] Compiling Test.Framework.HUnitWrapper ( Test/Framework/HUnitWrapper.hs, dist/build/Test/Framework/HUnitWrapper.o )
[19 of 19] Compiling Test.Framework   ( Test/Framework.hs, dist/build/Test/Framework.o )
Linking...
/usr/bin/ar -r dist/build/libHSHTF-0.11.2.a dist/build/Test/Framework.o dist/build/Test/Framework/HUnitWrapper.o dist/build/Test/Framework/TestManager.o dist/build/Test/Framework/TestTypes.o dist/build/Test/Framework/TestReporter.o dist/build/Test/Framework/CmdlineOptions.o dist/build/Test/Framework/QuickCheckWrapper.o dist/build/Test/Framework/BlackBoxTest.o dist/build/Test/Framework/Location.o dist/build/Test/Framework/Tutorial.o dist/build/Test/Framework/Pretty.o dist/build/Test/Framework/JsonOutput.o dist/build/Test/Framework/XmlOutput.o dist/build/Test/Framework/ThreadPool.o dist/build/Test/Framework/TestManagerInternal.o dist/build/Test/Framework/Utils.o dist/build/Test/Framework/Colors.o dist/build/Test/Framework/Diff.o dist/build/Test/Framework/Process.o
/usr/bin/ar: creating dist/build/libHSHTF-0.11.2.a
In-place registering HTF-0.11.2...
'/opt/ghc-7.4.2/bin/ghc-pkg' 'update' '-' '--global' '--no-user-package-conf' '--package-conf=/var/lib/jenkins/workspace/Stackage with GHC 7.4.2 no HP/stackage/sandbox/package-db' '--package-conf=dist/package.conf.inplace'
Preprocessing executable 'htfpp' for HTF-0.11.2...
Building executable htfpp...
creating dist/build/htfpp
creating dist/build/htfpp/htfpp-tmp
'/opt/ghc-7.4.2/bin/ghc' '--info'
'/opt/ghc-7.4.2/bin/ghc' '--info'
/opt/ghc-7.4.2/bin/ghc --make -no-link -fbuilding-cabal-package -O -static -outputdir dist/build/htfpp/htfpp-tmp -odir dist/build/htfpp/htfpp-tmp -hidir dist/build/htfpp/htfpp-tmp -stubdir dist/build/htfpp/htfpp-tmp -i -idist/build/htfpp/htfpp-tmp -i. -idist/build/autogen -Idist/build/autogen -Idist/build/htfpp/htfpp-tmp -optP-include -optPdist/build/autogen/cabal_macros.h -hide-all-packages -no-user-package-conf -package-conf /var/lib/jenkins/workspace/Stackage with GHC 7.4.2 no HP/stackage/sandbox/package-db -package-conf dist/package.conf.inplace -package-id array-0.4.0.0-0b6c5ca7e879a14d110ca4c001dd9297 -package-id base-4.5.1.0-66f22db3dfcd87541c9c7e50e7095d26 -package-id cpphs-1.18-2d03a5f4515ccd0d6e309a45cd973fdc -package-id directory-1.1.0.2-90bfc47dbd5ac1313aec9ce76cc22338 -package-id haskell-src-exts-1.14.0.1-fc3b421879fe65ce65174a95c15c086f -package-id old-time-1.1.0.0-703543375fafbcbe67bc51a758e84f10 -package-id random-1.0.1.1-c7787dee61cd418f6edd479355fe381c -package-id text-0.11.3.1-41f05eb413a8fa91f09d91d3051e4ae4 -XHaskell2010 ./HTFPP.hs
[1 of 6] Compiling Test.Framework.Utils ( Test/Framework/Utils.hs, dist/build/htfpp/htfpp-tmp/Test/Framework/Utils.o )
[2 of 6] Compiling Test.Framework.Location ( Test/Framework/Location.hs, dist/build/htfpp/htfpp-tmp/Test/Framework/Location.o )
[3 of 6] Compiling Test.Framework.HaskellParser ( Test/Framework/HaskellParser.hs, dist/build/htfpp/htfpp-tmp/Test/Framework/HaskellParser.o )

Test/Framework/HaskellParser.hs:82:13:
    Ambiguous occurrence `catch'
    It could refer to either `Prelude.catch',
                             imported from `Prelude' at Test/Framework/HaskellParser.hs:18:8-35
                             (and originally defined in `System.IO.Error')
                          or `Control.Exception.catch',
                             imported from `Control.Exception' at Test/Framework/HaskellParser.hs:23:38-42
                             (and originally defined in `Control.Exception.Base')

HTF misses good documentation. See also this "question" on StackOverflow.

When I try to use gassertFailure in a source file processed by htfpp, it doesn't expand and thus generates a compiler error:

Not in scope: ‘gassertFailure’
Perhaps you meant one of these:
  ‘gassertFailure_’ (imported from Test.Framework),
  ‘assertFailure_’ (imported from Test.Framework),
  data constructor ‘WaiTestFailure’ (imported from Network.Wai.Test)

In Test.Framework.Preprocessor, I see that allAsserts is missing assertFailure but it's patched in later, after adding the g variants.

As an aside, the reason I'm using the generic assertions is not for a pure result, but rather I want to use them within a monad stack which instantiates MonadIO. So I defined this orphan instance:

instance AssertM (ReaderT r IO) where
  genericAssertFailure__ loc s = liftIO $ genericAssertFailure__ loc s
  genericSubAssert loc ms act = do
    r <- ask
    liftIO $ genericSubAssert loc ms $ runReaderT act r

Would there be any interest in supporting MonadIO explicitly, either by defining similar AssertM instances, or by generalizing the regular assert* functions using MonadIO?

Hi,

sorry to open this as an issue here.

But I just don't manage to change the number of test runs
for QuickCheck.

My main looks like the following:

main :: IO ()
main = do setDefaultArgs $ stdArgs { maxSuccess = 2000 }
          htfMain htf_thisModulesTests

But the maxSuccess value has no effect.

I will try now to use withQCArgs, but it would be nice to be able
to change the default value.

Best
Kilian

Module Test.Framework.HUnitWrapper exports a lot of assertion functions (assertEqual_, assertBool_), but some of them has weird type signature. Say I want to implement test that always fails. I expect next code snippet will do what I want:

import Test.Framework
import Test.Framework.TestTypes

myFailedTest :: Assertion
myFailedTest = assertFailure

But assertFailure has signature assertFailure_ :: Location -> String -> IO a. Why ? Is it for internal purposes only ?

Currently when I run a test suite I get a message like Test suite internal-tests: RUNNING... and no output of the traced messages or logging until the test-suite finishes, and even then, if it finishes with success, the only way to see the output is by viewing the log files.

Besides the fact that there's not much useful about such behaviour by itself and the pain it causes during debugging, when I need to see the output, it also causes troubles like inducing prolonged execution times for applications with much output. The last point is also a cause of another problem: the Travis CI terminates jobs that produce no output for 10 minutes.

As a reference here is a build, that suffers from the mentioned issues. The source code is linked from there.

To resolve the issue I suggest to disable such behaviour of HTF, or make it optional at least.

• BlackBoxTests should be named ShellBasedTests
• it should be possible to write tests that only care about the exit code
• configuration mechanism should be simpler

The single quotes on this line make the HTF preprocessor ignore all the following tests. As a proof of a bug, a commit inserting a single character between them fixes the issue.

HTF-0.12.2.2

I've tried and tried, but for the life of me I can't get HTF to install on osx (I'm on 10.9).

I tried installing via cabal version 0.11, 0.10, and 0.9, all with the same failure. I cloned the repo and tried building but also to no avail. I am getting the following output

From the HTF folder of the cloned repo:

> cabal install
Resolving dependencies...
Configuring HTF-0.11.0.1...

/var/folders/b4/m9gsxwt15s329sq82kc0h8yr0000gn/T/30224.c:1:12:
     warning: control reaches end of non-void function [-Wreturn-type]
int foo() {}
           ^
1 warning generated.
Building HTF-0.11.0.1...
Preprocessing library HTF-0.11.0.1...
/var/folders/b4/m9gsxwt15s329sq82kc0h8yr0000gn/T/ghc30267_0/ghc30267_0.hscpp: openBinaryFile: does not exist (No such file or directory)
Failed to install HTF-0.11.0.1
cabal: Error: some packages failed to install:
HTF-0.11.0.1 failed during the building phase. The exception was:
ExitFailure 1

Any ideas? I would really love to get this to work

Data.Algorithm.Diff now uses First, Second, Both, instead of F, S and B data constructors. Also, your pattern matching against Both must change. I will send you a proposed pull request for this shortly.