nick8325 / quickcheck Goto Github PK

Steps to reproduce

-- file Main.hs
import Test.QuickCheck
import Test.QuickCheck.State
import Test.QuickCheck.Property

main :: IO ()
main = quickCheckWithResult args (callback progress prop) >>= print
  where
    prop :: [Int] -> Bool
    prop xs = (reverse . reverse) xs == xs

    args = stdArgs {maxSuccess = 100000, chatty = False}

    progress = PostTest NotCounterexample $ \st _ -> (print $ numSuccessTests st)

$ ghci Main.hs
*Main> :main

While the test is running, press ctrl-c.

Expected result:

• The program is terminated immediately (the UserInterrupt exception is propagated, so that the default exception handler kicks in)

Actual result:

With high probability

• the test keeps running
• the result is Success, with output set to "*** Exception in callback: user interrupt\n+++ OK, passed 100000 tests.\n"

quickCheck $ forAll arbitrary $ \b -> cover b 100 "a" True
+++ OK, passed 100 tests (only 68% a; not 100%).

forAll arbitrary $ \b -> conjoin [ cover b 100 "a" True ]
+++ OK, passed 100 tests.

I believe the conjoin case should also print a warning.

Just a question of understanding. We currently have:

instance (Arbitrary a, CoArbitrary b) => CoArbitrary (a -> b) where
  coarbitrary f gen =
    do xs <- arbitrary
       coarbitrary (map f xs) gen

Why do we do map f xs here? Why not simply:

instance (Arbitrary a, CoArbitrary b) => CoArbitrary (a -> b) where
  coarbitrary f gen =
    do x <- arbitrary
       coarbitrary (f x) gen

Thanks!

If we have a property that uses (===) and fails with an exception, e.g.

prop = (=== (undefined :: Int))

then quickCheckWithResult will leak that exception as part of the output when run with chatty = False.

Here are two GHCi sessions that demonstrate the issue. Note that for the first case (when chatty = True) the output field is a total string. In contrast, for the second case (when chatty = False) the output field contains an error thunk.

ghci> output <$> quickCheckWithResult stdArgs {chatty = True} prop
"*** Failed! Exception: 'Prelude.undefined' (after 1 test): \n0\nException thrown by generator: 'Prelude.undefined'\n"

ghci> output <$> quickCheckWithResult stdArgs {chatty = False} prop
"*** Failed! Exception: 'Prelude.undefined' (after 1 test): \n0\n0 /= *** Exception: Prelude.undefined

Already asked this on Stackoverflow AND Reddit, but no one seems to know.

I've been trying to solve the problem for the past 3 days now, but no matter what I try, the types don't compose elegantly.

Running quickCheckWith stdArgs { maxSuccess = a_big_number } (\() -> True) consumes more and more memory. The culprit seems to be the terminal module - strings which are output with putTemp are appended to the captured output even after they've been erased from the screen.

Ideally the exception handling logic would be based off of either the exceptions or the monad-control package, but even a simple try :: (Exception e, Monad m) => PropertyM m a -> PropertyM m (Either e a) would be satisfactory.

I took a stab at each of these myself, but I couldn't get the types to work out. I'm creating this issue in hopes that there's something I missed that will be obvious to someone more familiar with QuickCheck's inner workings.

The new release candidate for GHC 8.x cannot compile QuickCheck because of the following restrictions on template-haskell:

template-haskell >=2.4 && <2.11

This issue prevents of from building (or rather: testing) pretty much any significant package with that compiler. Would it be possible to get an update for QuickCheck that support the 8.x series of GHC?

Quick Check 2.8.2 does not work with any template-haskell. In particular, it does not work with version 2.11. Please take time and specify package version bounds.

I'm wondering if it would be possible to add an Alternative instance for Gen

The starting point for the change would be to move from:
newtype Gen a = MkGen { unGen :: QCGen -> Int -> a }
to either:
newtype Gen a = MkGen { unGen :: QCGen -> Int -> Maybe a }
or:
newtype Gen a = MkGen { unGen :: QCGen -> Int -> [a] }

I think the version with the list for the results would probably work better for <|>, with something like oneof being used to get back to a single value - like the current behaviour - when the list is non-empty.

While making a breaking change like that, I'd also be tempted to add another Int parameter in there, to be used to make <|> behave more like frequency, with corresponding helpers like sized / resize to help out.

I'm keen to give this a go myself and see how far I can push it, but I thought I'd put something here and so whether folks are strongly for or strongly against the idea, or if there are any landmines here I should be aware of, before I put too much time into it.

To make Arbitrary instances for my types I have to include quickcheck in my main library depends, rather than just testing, or deal with orphaned instance warnings by defining it in the test code only (and then I can't put properties in doctest, as well, I think).

I've got a quickcheck property that times out (5 second timeout) when the test case gets large enough. If I run that specific test case through the functions under test within GHCI, it completes instantly. Test failure is reproducible, happening after about 90 test cases every time.

You should be able to reproduce it with this file.

GHC version:

> ghc -v
Glasgow Haskell Compiler, Version 7.10.1, stage 2 booted by GHC version 7.8.4

If you want a cabal file, here's a chunk:

test-suite test
  default-language:
    Haskell2010
  type:
    exitcode-stdio-1.0
  hs-source-dirs:
    Test
  main-is:
    test.hs
  default-extensions: 
  build-depends:
      base >= 4.8
    , QuickCheck
    , bytestring

An example failure is

*** Failed! Exception: '<<timeout>>' (after 90 tests):  
Large {getLarge = 69071890884779960}

When I pass it through GHCI, it completes instantly, whereas QuickCheck says the test times out, taking >= 5 seconds.

> fromSignedByteString . toSignedByteString $ 69071890884779960
69071890884779960

It makes no progress if I remove the timeout and just let it run

> time dist/build/test/test
^C*** Failed! Exception: 'user interrupt' (after 94 tests): 
Large {getLarge = 102007353455633744}
182.37user

Memory usage doesn't climb, so it doesn't seem to be a large space leak.

Since I can't reproduce the problem when running my functions outside of quickcheck, I'm assuming that quickcheck is the problem.

The following are particularly notably missing: Word, Word16, Word32, Word64, Int8, Int16, Int32, Int64, Natural, Complex, Data.Sequence.Seq, Data.IntMap.IntMap, Data.Set.Set, Data.IntSet.IntSet, Data.Tree.Tree, Data.Array.Array, Data.Array.Unboxed.UArray, Text.PrettyPrint.Doc, Text.PrettyPrint.Annotated.Doc.

If I run verboseCheck prop with prop defined below, it slows my computer down and doesn't terminate. If I cancel the running command with Ctrl+c Ctrl+c, the command stops and prompt returns, but it seems process is still running and computer is slowed. Until I exit ghci prompt. This problem does not occur for quickCheck prop.

prop :: Fun String Integer -> Bool
prop (Fun _ f) = f "monkey" == f "banana" || f "banana" == f "elephant"

Applies to ver. 2.8.1

{-# LANGUAGE TemplateHaskell #-}
import Test.QuickCheck
import Test.QuickCheck.All
{-
prop_old_freeVars_Term conf x = True
-}
-- Template Haskell hack to make the following $quickCheckAll work under ghc-7.8.
return [] 
-- | All tests as collected by 'quickCheckAll'.
tests :: IO Bool
tests = do
  putStrLn "Agda.TypeChecking.Free.Tests"
  $quickCheckAll

gives warning

/home/abel/agda/src/full/Agda/TypeChecking/Free/Tests.hs:129:3: Warning:
    Name prop_old_freeVars_Term found in source file but was not in scope

which turns into error with -Werror

@jtdaugherty reports that somewhere between QuickCheck 2.8.2 and 2.9.1 a bug (or a silent and undocumented in the Changelog change in semantics) was introduced: UnkindPartition/tasty#147

Could you look into it?

import Test.QuickCheck

main = quickCheck $
  forAll arbitrary $ \b ->
    cover b 100 "b is True" $
      b ==> True

I would expect this to tell me something like b is true onl 50% of the time, but I get OK, passed 100 tests.

It seems odd to me that the inner ==> can sneak around the outer cover.

If I had written

import Test.QuickCheck

main = quickCheck $
  forAll arbitrary $ \b ->
    b ==>
      cover b 100 "b is True" $
        True

then that output would be expected, but in my case above cover comes first, so it surprised me.

Is this intended?

If so, could we make it clear in the documentation, maybe even this exact example?

I found this issue while preparing Agda for supporting GHC 7.10.1. The following example compiles with GHC 7.8.4:

{-# LANGUAGE TemplateHaskell #-}

import Test.QuickCheck

prop_id :: Eq a => a -> Bool
prop_id x = x == x

return []

main :: IO Bool
main = $quickCheckAll

Using GHC 7.10.1 RC 1, I got the following error:

$ ghc Test.hs
Test.hs:12:8:
    Exception when trying to run compile-time code:
      Higher-kinded type variables in type: forall (a_0 :: *) . GHC.Classes.Eq a_0 => a_0 -> GHC.Types.Bool
    Code: quickCheckAll
    In the splice: $quickCheckAll

monomorphic always uses VarE internally, which fails if it's given the Name of a constructor. For example, to make monomorphic lists:

{-# LANGUAGE TemplateHaskell #-}
import Test.QuickCheck.All

nil  = $(Test.QuickCheck.All.monomorphic '[])
cons = $(Test.QuickCheck.All.monomorphic '(:))

Loading this in GHCi gives:

    Illegal variable name: ‘[]’
    When splicing a TH expression:
      GHC.Types.[] :: [GHC.Integer.Type.Integer]
    In the splice: $(Test.QuickCheck.All.monomorphic '[])
Failed, modules loaded: none.

Using ConE would allow constructors (after branching on whether the Name could be a variable or not).

QuickCheck depends on containers, could the Arbitrary instances for it be added?

Naive Map generation might not be what you want, but it's still better then no instance. Set and Seq should be trivial.

I can write a PR, if the idea is ok.

We use cover to ensure that our tests are meaningful (e.g. not vacuously true due to ==> or that cases we care about are actually generated).

Currently cover is only informative, so if a cover is false you get some warning printed out by quickCheck, but the test still passes. If you use a test runner like test-framework, tasty or hspec, they do not notice.

Could we have a version of cover that strictly lets the test fail?

This would make it possible to notice when somebody changes implementation code that lets our test cases degenerate.

GHC 7.10 will ship with base-4.8.0.0 which will have the Natural data type. It'd be great if QuickCheck came with a Arbitrary instance.
Also see kolmodin/binary#63.

I tried to make a matrix with GHC 6.12.3, and it starts to be quite troublesome to compile. Maybe one could get further by disabling tests, but then tested-with would lie.

Particularly, seems that I broke binary-0.8 for GHC <7.0 (because it isn't tested)

EDIT this would highly simplify cabal file too.

Congratulations on the release

This commit adds a Boolean but it is unclear what is being tested when I see not b, why not use an ad-hoc type instead? I need help coming up with a name though :)

data HaveIBeenShrunk = UnShrunken | Shrunken

data Honey_I = Didn'tShrinkTheKids | ShrunkTheKids

Hi,

first many thanks for opening the issue tracker. I think this will benefit QuickCheck a lot.

Now to my issue. Note that it is written with more information than necessary for the QuickCheck maintainer (most of it is probably immediately clear to him) because I wanted to explain the situation to those who haven't spent the last 10 hours on the code.

Currently (as of commit 205f9f9), genericShrink (0::Int) == [0]. I don't think this is intended, since the documentation of shrink mentions that shrinking a value to itself will result in an infinite loop.

The problem is this:

instance Typeable a => Subterms (K1 i a) where
  gsubterms (K1 x) =
    case cast x of
      Nothing -> []
      Just y -> [y] -- Bug: this will shrink 0 to [0]

I first believed that it can be fixed by Just y -> shrink y, but that is not so because then genericShrink [0::Int] == [], which is also wrong, it should be [[]].

I noticed that 2 years ago, @bitonic created a gShrink/gArbitrary implementation (see here that doesn't use the Typeable constraint, but instead relies on OverlappingInstances. In my opinion this is better because it removes a constraint from genericShrink, requiring only Generic, and the OverlappingInstances is only needed in Arbitrary.hs to cleanly select the current type and has no effect outside of this file. And of course, it does not come with a run-time penalty while Typeable does.

I have created a branch that compares the 5 different ways that I've seen so far - have a look here:

1. genericShrinkOrig - the original genericShrink implementation. Wrong because it shrinks 0 to 0.
2. genericShrinkOrigFixed - my try to fix it by using shrink in the base case. Wrong because it shrinks [0] to [].
3. genericShrinkBitonic - same output as subtermsOrig (so recursively shrinking subterms is missing) and having the same problem as 1.
4. genericShrinkBitonicFixed - my try to fix it by using shrink in the base case. Same output as subtermsOrigFixed (so recursively shrinking subterms is missing) and having the same problem as 3.
5. genericShrink, my improved version, is the apparent solution (please check thorougly if what I say is correct). Explanation:

The core problem is to correctly capture the semantics of shrink in the generic instance, that being the points:

1. Shrink a term to any of its immediate subterms.
2. Recursively apply shrink to all immediate subterms.
3. Type-specific shrinkings such as replacing a constructor by a simpler constructor.

Let's go through them, on the example of data Tree a = Nil | Branch a (Tree a) (Tree a):

1. Implemented by subterms. What are the immediate subterms of Tree a that are of the same type Tree a (since we can only shrink to values of the same type)? The left and the right branch. Not the value a, because it is not a tree. This is what the Typeable is for: GHC.Generics will give us all 3 subterms, [a, Tree a (left), Tree a (right)] separated by *:*, and we have to cast in order to check if each of them really is a Tree. In @bitonic's implementation, this check is done at run-time instead of at compile time by using class GShrinkBitonic f b instead of class SubtermsOrig f - note how there's one extra type parameter. This allows him to distinguish the two cases instance Arbitrary a => GShrinkBitonic (K1 i a) a (subterm is of same type as the term) and instance GShrinkBitonic (K1 i a) b (subterm is of different type) using OvelappingInstances. Back to the problem: In the Subterms (K1 i a) case, the case x of Just y -> [y] returns [0] when run on 0::Int because the generic instance of Int is basically a K1 around itself: from (1::Int) == M1 {unM1 = M1 {unM1 = M1 {unM1 = K1 {unK1 = 1}}}}.
2. Implemented by recursivelyShrink. The idea here is to keep the same outer structure, but shrink the inner structures. So basically Branch x l r becomes Branch x (shrink l) (shrink r), for any combination of shrinking the left and right part. This is done correctly by the current implementation. For sum types (RecursivelyShrink (f :+: g)) just do it inside, for product types (RecursivelyShrink (f :*: g)) recursively do it for left or right side inside the existing outer structure, for the base case RecursivelyShrink (K1 i a) do grecursivelyShrink (K1 x) = map K1 (shrink x), calling shrink on the values inside.
3. Clearly, type-specific shrinkings are not possible for a generic implementation, since that is for any type and thus can't be type-specific. Thismeans that shrinking to simpler constructors must always be done manually by the human if some constructors are "simpler" than others (like Nothing and Nil), and that genericShrink can only be used for doing all the non-specific cases, as is explained with shrink x = shrinkToNil x ++ genericShrink x in the QuickCheck docs. For types where the constructors are equally simple, a shrink = genericShrink is enough.

So the problem is in 1, namely that subterms creates things that are not subterms (0 isn't a subterm of 0).

Looking at 1, my *Fixed versions are wrong, since I call shrink inside of subterms, and "Shrink a term to any of its immediate subterms" really means "just take the inner terms of your structure, as they are".

So what's the right subterms?

A value can only have true _sub_terms if it has a product type (:*:) somewhere.

• If it doesn't, we return [].
• If it does, we return the first value we can grab on either side of the (:*:).

That means we need two type classes for this: An outer GSubterms and an inner GSubtermsIncl(usive).

• The outer one noticeably returns gSubterms (K1 _) = [], and on products calls the inner one: gSubterms (l :*: r) = gSubtermsIncl l ++ gSubtermsIncl r.
• The inner (inclusive) one returns everything it can grab, namely gSubtermsIncl (K1 x) = [x] if the type matches (this can be done with GSubtermsIncl (K1 i a) a or Typeable).

As a result, subterms returns the first container in a value, unless the value consists only and entirely of that container.

This gives the right result for all subterms invocations I've tried so far. Please check if you agree with my reasoning. Again, the working example is genericShrink in here.

I will soon submit a pull request with the cleaned-up version of this.

This package depends on array transitively, via containers, so adding it as a direct dependency can't do any harm.

How would we feel about adding the text package as a dependency of QuickCheck? I know it's good to keep QuickCheck's dependencies light, but I think depending on Haskell's recommended string type is a reasonable measure.

Right now there's probably an orphan Arbitrary Text instance in pretty much every sizable application that uses QuickCheck (and library authors have it even worse because they don't get that option).

Negative Gen sizes cause all sorts of trouble. Probably resize should just throw an error on negative sizes to highlight incorrect generators early during development/testing.

I was trying to figure out why generate (vector 100 :: Gen [Large Int]) was only giving me small domain values, and noticed that arbitrarySizedBoundedIntegral is incorrect:

>  fmap maximum $ replicateM 1000000 $ generate (arbitrarySizedBoundedIntegral :: Gen Int)
4096

That code's a bit opaque to me, sorry I'm not of much help.

Citing from http://hydra.cryp.to/build/1054975/nixlog/1/raw:

Test/QuickCheck/All.hs:76:11: error:
    The constructor ‘ClassOpI’ should have 3 arguments, but has been given 4
    In the pattern: ClassOpI _ ty _ _
    In an equation for ‘infoType’: infoType (ClassOpI _ ty _ _) = ty

I think this is probably sensible:

instance Arbitrary QCGen where
  arbitrary = MkGen (\g _ -> g)

Zemyla (jokingly?) suggested also making it an instance of CoArbitrary so that Gen can also be made an instance of Arbitrary. Why not?

instance CoArbitrary QCGen where
  coarbitrary s = variant (fst (next s))

With this in place, a property can ask for a generator and do with it what it likes.

It would be nice to be able to use existential quantification in QuickCheck.

Since ∃x(φ(x)) is equivalent to ¬∀x(¬φ(x)), the easiest thing to do would be add an invert :: Testable prop => prop -> Property "Property Combinator".

In the instance for Arbitrary (Positive a) the definition for arbitrary is this:

arbitrary =
    ((Positive . abs) `fmap` (arbitrary `suchThat` (/= 0))) `suchThat` gt0
    where gt0 (Positive x) = x > 0

Is the check ``suchThat gt0 just in case the `Num` instance for `a` has a misbehaving definition for `abs`?

Recently I found out QC never finds counterexample to following property even after 1e8 test cases:

p_failing :: Word64 -> Bool
p_failing i = testBit i 63 == False

But it's false for exactly half of Word64! Then plotted frequency for each bit of Word64 which is passed to property (1e6 test cases). Bits 63 and 62 are always zero in all test cases.

And looking at consecutive inputs reveals strange pattern:

     1  0x0000000000000001
     2  0x0000000000000000
     3  0x0000000000000001
     4  0x0000000000000000
     5  0x0000000000000002
     6  0x0000000000000004
     7  0x0000000000000005
     8  0x000000000000000b
     9  0x000000000000000c
    10  0x0000000000000009
....
    90  0x004fe9b98bdd482e
    91  0x00bf83bfb1a60c70
    92  0x00a8e49cc9fee6ca
    93  0x00385b6297fbde9d
    94  0x007069be7a8569e9
    95  0x06f7323a8211c169
    96  0x05668182d04237f2
    97  0x0ebac5a91be8d419
    98  0x134527f8393c2980
    99  0x1f2b0200977d9706
   100  0x3a1ce8ef8c3eac26
   101  0x0000000000000000
   102  0x0000000000000000
   103  0x0000000000000000
   104  0x0000000000000000
   105  0x0000000000000004

Every 100 of test cases generator starts to explore search space from small numbers again and never reaches big Word64

Hi all -

I'm having trouble with QuickCheck reporting correct examples as failing. I've put together a minimal example demonstrating this behavior in a Gist. I've trimmed it down and tested it on multiple machines, and I believe by this point that it's a library bug.

The following is copied for your convenience from the README in the Gist:

QuickCheck Bug?

This repo documents a strange behavior I encountered when trying to test some code using the Data.Graph.Inductive library with QuickCheck.

QuickCheck will report failing examples for the test mergeWithFixTrue (see Test.hs). However, when these failing examples are copied into unit tests and run using HUnit, they invariably succeed.

This begs the question - why QuickCheck? Might HUnit not as easily be at fault? There are two reasons why I believe QuickCheck might be at fault here:

1. Looking at small failing examples and working the function call out by hand, the failing examples should not be failing.
2. By inserting calls to Debug.Trace.trace, I've been able to tell that mergeWith (see Merge.hs) behaves differently when called with fixBody from QuickCheck vs. HUnit or cabal repl. When it is called via QuickCheck, mergeWith fixBody returns a graph with an extra edge in it, and it is this unwanted edge that causes the test to fail.

Running

To reproduce the behavior described above, clone this repository and initialize a cabal sandbox with cabal sandbox init. Then run the following:

1. cabal install --only-dependencies --enable-tests
2. cabal test

You may have to run cabal test a few times to get a failing example from QuickCheck. I like to run cabal test until I get a nice, bite-sized failing example (no more than 3 or 4 edges in the example graph).

Once you've done this, insert the failing example into the HUnit test in Test.hs on the line starting with: "testCase" (the failing example should go in the let-binding: let g = <failing example here>).

Run cabal test again, and notice that the example which QuickCheck reports as a failure runs perfectly well in HUnit.

Hello,

It would be nice to have a built-in generator for randomly ordering a list.

Something along these lines:

import System.Random.Shuffle (shuffle)

-- | Take a list and generate a shuffled version of it.
shuffled ::[a] -> Gen [a]
shuffled xs = do
  rs <- randomOrdering (length xs - 1)
  return $ shuffle xs rs
  where
    -- a sequence (r1,...r[n-1]) of numbers such that r[i] is an independent
    -- sample from a uniform random distribution [0..n-i]
    randomOrdering 0 = return []
    randomOrdering n =
      do y <- choose (0, n)
         ys <- randomOrdering (n - 1)
         return (y:ys)

Depends on System.Random.Shuffle

Like an http://hackage.haskell.org/package/transformers-0.5.1.0/docs/Data-Functor-Classes.html

I can make an PR if this idea is OK

We have a few tests that are hitting the 4MB log limit on Travis because all the intermediate progress is logged out by QuickCheck. We have a temporary fix which processes \b characters before outputting a line, but it would be useful if we could just pass an option to QuickCheck to provide slightly quieter output.

Happy to do the work if you have some idea how you'd like it done.

The following are used in containers

apply2 :: Fun (a, b) c -> a -> b -> c
apply2 f a b = apply f (a, b)

apply3 :: Fun (a, b, c) d -> a -> b -> c -> d
apply3 f a b c = apply f (a, b, c)

I see two directions we can go. We can supply those functions, along with:

pattern Fn :: (a -> b) -> Fun a b
pattern Fn f <- Fun _ f

pattern Fn2 :: (a -> b -> c) -> Fun (a, b) c
pattern Fn2 f <- Fun _ (curry -> f)

pattern Fn3 :: (a -> b -> c -> d) -> Fun (a, b, c) d
pattern Fn3 f <- Fun _ (curry3 -> f)

curry3 f a b c = f (a, b, c)

now it can be written

pInsertWithKeyStrict :: Fun (Int, Int) Int -> Int -> IntMap Int -> Bool
pInsertWithKeyStrict (Fn2 f) v m = isBottom $ M.insertWith f bottom v m

pInsertLookupWithKeyKeyStrict :: Fun (Int, Int, Int) Int -> Int -> IntMap Int -> Bool
pInsertLookupWithKeyKeyStrict (Fn3 f) v m = isBottom $ M.insertLookupWithKey f bottom v m

Alternatively we could use some type class trickery to provide a single function and pattern that supersede apply{,2,3} and Fn{,2,3}. So that the user never has to think about how many arguments their function has.

I have some doubts about the effects that will have on inference.

One last solution: somehow define an Arbitrary (Fun a (b -> c)) instance. I have not thought this through but if this makes sense it might be the most elegant solution.

Generate edge cases (minBound, maxBound, minBound + 1, ...) with some low frequency.

abs always returns a positive number right?

>>> quickCheck (\(n :: Int) -> abs n >= 0)
+++ OK, passed 100 tests.

Alright let's ship it — well — the test fails if we run it on Int8s often enough

>>> quickCheck (\(n :: Int8) -> abs n >= 0)
+++ OK, passed 100 tests.
>>> quickCheck (\(n :: Int8) -> abs n >= 0)
+++ OK, passed 100 tests.
>>> quickCheck (\(n :: Int8) -> abs n >= 0)
+++ OK, passed 100 tests.
>>> quickCheck (\(n :: Int8) -> abs n >= 0)
*** Failed! Falsifiable (after 23 tests):                  
-128

Turns out it doesn't actually hold for Ints

>>> abs @Int minBound
-9223372036854775808

This could be detected for larger sample spaces like Int if minBound were generated. I would like to see this in the Arbitrary Int instance but maybe a modifier is better?

When running with Git HEAD:

QuickCheck-2.9.1: test (suite: test-quickcheck)

=== prop_ToSortedList from examples/Heap.hs:123 ===
*** Failed! Falsifiable (after 7 tests):  
Node (-4) (Node 4 (Node 7 Empty Empty) (Node 8 Empty Empty)) (Node (-2) (Node 5 Empty Empty) (Node (-1) Empty Empty))

QuickCheck-2.9.1: test (suite: test-quickcheck-gcoarbitrary)

*** Failed! Falsifiable (after 4 tests and 4 shrinks):    
{_->2}

Commit e0607bf removed the Integral and Real instances for Positive, NonZero and NonNegative.

Two questions:

• Why were the Real instances removed? They only provide toRational :: a -> Rational, which cannot break any invariants.
• That the Integral instances for Positive and NonZero were removed makes sense since with mod n n we could create create a forbidden 0. But for NonNegative, 0 is allowed. Could its Integral instance be reintroduced?

This is just a little feature request so I don't forget it. If I end up implementing it I'll report back or do a PR if this would be useful.

The idea is if we're testing functions that make use of some bit twiddling, operate on bytes of a word, etc. we might like an Arbitrary instance for instances of Data.Bits.Bits that implements a shrink by perhaps zeroing individual bytes of a word, and then maybe zeroing individual bits (or something; I haven't thought much about this yet).

I saw this SO post https://stackoverflow.com/questions/8191131/find-the-value-that-failed-for-quickcheck

is it possible to have a function like this build into quickcheck itself?

My use case is that i have a parser and a pretty printer and the test flow is like this:

quickcheck generate random syntax tree -> print -> source code -> parse -> syntax tree (should be same as input)

If a test case fails i get the syntax tree which is hard to read and i like to try to send it to the printer.

I'm pretty sure:

instance Testable Property where
  property = property . unProperty

Should have a definition of exhaustive. Currently:

Test.QuickCheck> quickCheck $ 1 == 1
+++ OK, passed 1 tests.
Test.QuickCheck> quickCheck $ property $ 1 == 1
+++ OK, passed 100 tests.

I think adding the line:

exhaustive = exhaustive . unProperty

Would do.

I found myself wanting this today.

class Monad m => MonadGen m where
  liftGen :: Gen a -> m a

instance MonadGen Gen where
  liftGen = id

instance MonadGen m => MonadGen (StateT s m) where
  liftGen = lift . liftGen

-- and so on

Similar to dreixel/generic-deriving#41, QuickCheck doesn't build with template-haskell-2.11.0.0 on GHC-7.8.4 (and GHC-7.10.3). Cabal produces install plans which attempt to do so.

I will provide a simple cabal file to reproduce the issue.

I want to test a property with a limited range of values, say the characters from 'a' to 'z'. I create a custom Gen for this purpose and run quickCheck on it. Unfortunately, if a test fails the failing arguments are not reported:

QC> QC.quickCheck $ fmap (<'f') $ QC.choose ('a','z')
*** Failed! Falsifiable (after 1 test): 

Is it possible to add this report? If not, how about an alternative form of quickCheck that better supports Gen parameters? E.g. an alternative form might be called like this:

QC> QC.quickCheckCustom (QC.choose ('a','z')) $ \c -> c<'f'
*** Failed! Falsifiable (after 1 test): 
't'

If you write with cover: "I want that x% (e.g. 100%) of my generated characters are a", and no as are generated at all, cover doesn't warn you when it should.

Example 1:

quickCheck $ forAll (return 'b') $ \c -> cover (c == 'a') 100 "a" True
+++ OK, passed 100 tests.

The problem is that quickcheck only considers stamps that actually occur.

This also happens if the percentage is very small so that with high probability no suitable sample is found; example 2:

> quickCheck $ forAll arbitrary $ \c -> cover (c == 'a') 100 "a" True
+++ OK, passed 100 tests (only 1% a; not 100%).
> quickCheck $ forAll arbitrary $ \c -> cover (c == 'a') 100 "a" True
+++ OK, passed 100 tests.

The location where cover is handled is covers in success :: State -> IO () in QuickCheck.Test.

We have quite a lot of quickcheck properties with replay arguments. The replay arguments are specified with the old StdGen. However, the most recent quickcheck version forces you to use tf-random.

Would it be possible to have a cabal flag specifying that you want to use StdGen? For older versions of ghc and base, you do this anyway.