A Testing Framework for Haskell
Home Page: https://hspec.github.io/
License: MIT License
Read the User's Manual!
Preparing a release:
$ util/release
or
$ util/release major
This will bump the version and update CHANGES.markdown.
Releases happen automatically when a new version ends up on main.
Whenever .github/workflows/publish.yml detects
a new version without a corresponding tag it:
[verbose] skipping test/main.hs because only library modules (like "module Main where") can be dynamically loaded.
no valid *.hs files found
I have a bunch of pragmas at the top of my file
{-# OPTIONS_GHC -fno-warn-unused-binds #-}
{-# LANGUAGE QuasiQuotes #-}
Organizations have have multiple owners with full access. This should make it easier for people to contribute and manage, especially since I haven't had as much time for Haskel lately and admin stuff isn't my strong suit.
It looks like the current url will become invalid and there won't even be a redirect. I've already set up the organization but just wanted to double check before moving it.
Should hspec be moved from www.github.com/trystan/hspec to www.github.com/hspec?
The documentation already says, that they are deprecated. But to provide a smoother upgrade path, they will only be DEPRECATED in a future release.
I'm trying to debug some monadic code, but HSpec seems to be suppressing any "putStrLn" that I put into the code.
This doesn't make sense to me. Why is suppression necessary? Why does HSpec prevent me from debugging the problem via putStr? How should I be debugging if putStrLn won't work? (AFAICT the documentation doesn't mention how you'd actually find the problem if a HSpec test fails.)
Hi,
I'd like to use travis-ci.org for my project http-types, but unfortunately it failed, complaining about Control.Monad.Trans.Writer. Downgrading to hspec 0.6.1 fixed the problem, but I'd like to use a newer version if possible.
/tmp/hspec-0.9.1.12104/hspec-0.9.1.1/Test/Hspec/Monadic.hs:85:8:
Could not find module `Control.Monad.Trans.Writer':
It is a member of the hidden package `transformers-0.2.2.0'.
The full build log should be here: http://travis-ci.org/#!/aristidb/http-types/builds/860905
It would be super cool if you could take a report like this:
myabs
and produce empty specs that look something like this:
specs :: IO Specs
specs = describe "myabs" [
it "returns the original number when given a positive input"
undefined,
it "returns a positive number when given a negative input"
undefined,
it "returns zero when given zero"
undefined]
I would like to insert print statements in my tests when they fail, but silently silences them.
Currently
specs = describe "foo" $ do
describe "bar" $ it "has some property" True
produces the following output:
bar
- has some property
Finished in 0.0000 seconds
1 example, 0 failures
I think it would be useful if it instead would produce:
foo
bar
- has some property
Finished in 0.0000 seconds
1 example, 0 failures
This would allow me to label specs, when combining them from several modules into a single spec.
Currently, only one level of describe is supported. I want something like describe "Application" $ describe "Module" $ describe "Submodule".
Currently I find myself writing describe "Module: Submodule", which seems inappropriate.
It seems like the rest results are made green, but the green is not reset afterwards, so now my console prompt is green too, until I run another command that does reset the colors.
I would like to use hspec, but I can't!
This is the install failure I experience. It is also listed on the hackage page for the package.
cabal: cannot configure hspec-0.2.0. It requires base >=4.2 && <=4.3
For the dependency on base >=4.2 && <=4.3 there are these packages:
base-4.2.0.0, base-4.2.0.1 and base-4.2.0.2. However none of them are
available.
base-4.2.0.0 was excluded because of the top level dependency base -any
base-4.2.0.1 was excluded because of the top level dependency base -any
base-4.2.0.2 was excluded because of the top level dependency base -any
I never really liked prop, because it does not really fit in when reading a specification.
spec = do
describe "The size of a widget" $ do
prop "is always greater or equal to 0" $ do
\widget -> 0 <= size widgetWould always be nicer?
spec = do
describe "The size of a widget" $ do
always "is greater or equal to 0" $ do
\widget -> 0 <= size widgetCurrently AnyExample is an existential; and I think we can express the same thing as an ordinary algebraic data type.
Something like this should work.
class Example a where
evaluateExample :: a -> AnyExample
data AnyExample = AnyExample (IO Result)
See also: http://lukepalmer.wordpress.com/2010/01/24/haskell-antipattern-existential-typeclass/
The issue is, that if you use hspec instead of hspecX in a cabal test suite, cabal test will not report failing tests.
This is not a theoretical issue, at least http-types does it wrong.
So I'd suggest to rename hspecX to hspec, and keep hspecX as an alias. In the long run I'd tend to first deprecate and later remove hspecX.
@trystan @gregwebs Are you aware of any use cases for hspec? If I understand correctly, it would theoretically allow you to re-create the report from an evaluate spec (possibly with a different formatter). But we have no code to support this; and the user could always just reevaluate the spec to achieve the same thing. Am I missing something?
BTW: From a discussion with @markwright (related to yesodweb/yesod@9f4c92f) I realized that the steps required to update a non-monadic spec for hspec-1.0 can in some cases be less obvious than I thought. I made some changes that will give better error messages (basically I newtype-wrapped UnevaluatedSpec). I'll push a 1.1 release that includes that change on Hackage today.
0.9.1 on hackage still has the Num instance problem
This is the only feature I am missing from the test-framework setup. Hopefully there is an existing package that handles any possible issues with terminal coloring.
BTW, Michael Snoyman is liking hspec also, so we may be adopting it for Yesod.
As far as I can tell, hspec always returns an exit code of 0, even though the tests fail. This makes it hard to use in a unixy environment.
What about re-exporting Test.Hspec.QuickCheck and Test.Hspec.HUnit from Test.Hspec and Test.Hspec.Monadic?
Pros:
-Wall-sane code)Test.Hspec.QuickCheck and Test.Hspec.HUnit from the public interface, which may make the documentation more accessible (this is backward compatible, if we only suppress Haddock documentation, but still export them)Cons:
property in scope (plus prop for the monadic interface), even if our spec does not contain any qc propertiesI'd be happy to supply a patch, if there is consent.
Say it would be nice if we could pipe the output of a test run to a file, like:
$ runhaskell Spec.hs > report.txt
Currently we use hspec to run hspec's own tests. I see two potential problems here.
One solution could be to use a previous release of hspec to run hspec's tests.
The stand-alone test runner is lacking these features, which I currently have from building my current test suite:
It is going to take a bit more work to make a stand alone test runner that can satisfy these demands, all of which are essential.
Perhaps the best short-term solution is to allow the runner to be built into a test suite. Essentially most of the code can be moved into the library. The stand-alone runner would just be a main function that calls that code, just as a stand-alone test suite can.
This will always provide an easy work-around for any problems with the runner until it is perfected.
It would be nice to be able to configure how much detail the hspec runner should give when running tests that don't fail.
If #16 is implemented, it would also be part of this request that one can specify how deep in the hierarchy the output should go.
I would like to be able to run individual it tests in their own database transaction (that gets rolled back). I would also like a before/setup method for common setup that will also get rolled back. #22 gets at 2 ways of doing this- immediate evaluation makes it possible to just put this burden back on the user, but that has other issues as discussed. It also makes describe a first class citizen- a describe could also have a before block.
My favorite ruby testing framework is riot. The setup method is more like a before(:all).
I am wondering if before/setup blocks are even possible in the non-monadic DSL in terms of using a value from that block in an it block. It seems that the it block would need to take an argument, so then you would need two different it methods.
Maybe the other technique is just
let setup = insert data in describe []
Sorry for rambling! We should discuss over a beer :).
It would be cool if we could selectively run our specs.
When building an app specs count gets higher and higher
and running all those tests again and again starts to take
more and more time.
Even simple export of Spec "name" function would do because
then we could manually filter all specs by - ex. cmd args..
Given the descriptive nature of hspec, this is a less pressing concern. Right now one can look at the string for the failure, and find that in the source. However this process could be automated to having hspec report a line number.
HTF provides line numbers by running a pre-processor that replaces assertions with assertions with line numbers. Perhaps that could be used directly. Otherwise, a simpler pre-processor for hspec could could just give line numbers for the 'it' string without figuring out the exact test failure line.
http://hackage.haskell.org/packages/archive/HTF/0.7.0.0/doc/html/Test-Framework-Tutorial.html
Here is a minimal example (aborts on "foo", "bar" will never be tried):
import Test.Hspec.Monadic
import Test.Hspec.QuickCheck
import Test.QuickCheck hiding (property)
data Foo = Foo
deriving (Eq, Show)
instance Arbitrary Foo
where arbitrary = undefined
main = hspec $ do
it "foo" $ property (== Foo)
it "bar" $ property True
The error message is
Resolving dependencies...
cabal: cannot configure hspec-0.8. It requires hspec ==0.7
There is no available version of hspec that satisfies ==0.7
Apparently the hspec.cabal file contains the line: build-depends: hspec == 0.7 which seems erroneous.
The haddock documentation doesn't show up on http://hackage.haskell.org/package/hspec
Since the documentation has many examples and has taken more time than anything else, it would be nice to have it available on hackage. Maybe it's something in the cabal file or an option when uploading to hackage?
(and use it to generate the reports given at hspec's website)
The specdoc formatter shows the reason for a failed test only after all tests are done. This has the disadvantage that (depending on how long your tests take) you may have to wait quite some time before you can start fixing a failing test.
I really like the current visual appearance of specdoc, but wasting programmer productivity is a no-go. So I think we have to do something about this.
The hspec DSL is already good, but I think it can be improved.
main = hspecX $ do
describe "hspec's monadic EDSL" $ do
it "has no commas" $ do -- "$ do" only necessary for multi-line test
describe "supports nested describes" $ do
hunit "appends describe text" (text @? "appended")
describe "supports multiple describes $ do
...
Having to explicitly evaluate the examples rather than rely on Haskell's laziness should allow us much more freedom in terms of when and where costly computations get run. This may also simplify the code and signatures since many things remained in the IO monad to avoid evaluating them.
(I've already done some insomnia-fueled hacking with this and will follow up on this issue when ready)
Test run is not abort on ctrl-c. Here is a minimal example, that reproduces the issue:
import Test.Hspec.Monadic
fib :: Int -> Int
fib 0 = 1
fib 1 = 1
fib n = fib (n - 1) + fib (n - 2)
main = hspec $ do
it "foo" $ fib 100 == 0
it "bar" $ fib 100 == 0
it "baz" $ fib 100 == 0
If you run this and press ctrl-c, only one test is abort, not the whole test run.
We had the same issue in HUnit, but this is fixed in 1.2.4.2.
I really like where this library is going, but there are still some fundamentals that need to be worked out.
On a test failure we just get the "it" description marked with an 'x'.
x is specified with the HUnit "TestCase" data constructor (example)
What we really need is specific information about the failed assertion in the output:
expected: (1,2)
but got: (1,3)
For this library to be compelling we need either
We currently only print the CPU time in the footer. This may be much smaller than the actual time passed (especially when you interact with external processes). I'd tend to print both, the total time passed, and the CPU time.
I'd write a patch, after that we can do the release.
The purpose is to aim for keeping the API stable over time. The internal part would still be exported, but with no guarantees for stability.
I think the API would just contain the Example type class and the Result type.
I just installed the 64 bit version of the haskell package for osx. Currently, a spec suite that ran in ~0.1s before now runs in:
Finished in 10826199509.0232 seconds
I'm pretty sure this is probably a bug with getCPUTime, but I thought you should know anyway.
The current descriptions (from Test.Hspec.Core) is more likely to confuse than help anyone using the monadic style. What users really want is do spec1; spec2 (assuming they are fond of monadic style) or sequence_ [spec1, spec2]. If you like to keep parity with the non-monadic API this could be exported:
descriptions :: [Specs] -> Specs
descriptions = sequence_
I could send the patch if you let me know what you prefer (hiding, changing, keeping as-is).
(analogous to RSpec's context)
*** Exception: getCurrentDirectory: resource exhausted (Too many open files)
I haven't tested if the same occurs when using the list API. Breaking the tests down into smaller calls to hspec rather than combining them all in one seems to help. My tests are all QuickCheck properties.
(keeping a deprecated synonym for backward compatibility)
HUnit distinguishes between errors and failures, hspec makes no such distinction. From a users perspective this has the disadvantage, that it is not immediately clear whether a failed test is due to an exception or a failed assertion. Here is an example:
import Test.Hspec.Monadic
import Test.Hspec.HUnit ()
import Test.HUnit
main :: IO ()
main = hspecX $ do
describe "a failing spec" $ do
it "treats a failed assertion as FAILED" $ do
assertBool "foobar" False
it "treats an exception as FAILED" $ do
error "foobar" :: Assertion
I find myself using comment blocks in hspec to just fail one test. This is already easier in hspec than other frameworks where you might have 2 distant places that need to be commented out (at least to avoid any compiler warnings).
When I use rspec and have a test failure, by default it will give both the line number and the description of the failed test. So I can run just the test with -l LINE_NUMBER or -e DESCRIPTION. Line number is quicker, but then if you make a drastic change in the tests it becomes incorrect. So I still use the description frequently.
There was already a request for the hspec test to parse the command line to give formatter options.
If the test suite generated by hspec will parse the command line, then we can add the -e capability.
Ok, here is an idea.
I often give one or more specific examples and a qc property for something that should hold. Here is an example:
describe "parse" $ do
it "is inverse to render" someSpecificExample
it "is inverse to render" someOtherSpecificExample
it "is inverse to render" property genericProperty
This produces the following output:
parse
- is inverse to render
- is inverse to render
- is inverse to render
What about producing the following output instead:
parse
- is inverse to render (3)
The primary goal is to save valuable screen estate and reduce clutter. If one of the cases fails, we could still give the expanded output.
Do you have any other ideas, how we could deal with this. Or are there other approaches to testing, that make the above irrelevant. I'd be very happy to hear about your thoughts on this.
I didn't know this because i don't use Windows, but Michael Snoyman just told me it is the case.
Is it ok for the executable be disabled by default on Windows for now? The long-term solution is probably to add a package hspec-runner or hspec-cl or something which just contains the exe.
I have an executable that runs all my specs, some of them written with hspec list descriptions, some of them written in the monadic style of Test.Hspec.Monadic.
Currently "concatenate them together" like this:
main = ListHspec.hspecX $ concat [
ModuleUsingListSpecs.specs
, runSpecM $ descriptions [
ModuleUsingMonadicSpecs.specs,
Module2UsingMonadicSpecs.specs
]
]However, I don't like this too much for two reasons:
runSpecM, which is commented as "internal only" in the hspec sourceModuleUsingListSpecs.specs in a nice way.
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.