bodigrim / smallcheck

Test your Haskell code by exhaustively checking its properties

SmallCheck: a property-based testing library for Haskell

As of 2023, this library is largely obsolete: arbitrary test generators with shrinking such as falsify offer much better user experience.

SmallCheck is a testing library that allows to verify properties for all test cases up to some depth. The test cases are generated automatically by SmallCheck.

Usefulness of such an approach to testing is based on the following observation:

If a program fails to meet its specification in some cases, it almost always fails in some simple case.

In many ways SmallCheck is very similar to QuickCheck. It uses the idea of type-based generators for test data, and the way testable properties are expressed is closely based on the QuickCheck approach. Like QuickCheck, SmallCheck tests whether properties hold for finite completely defined values at specific types, and reports counter-examples.

The big difference is that instead of using a sample of randomly generated values, SmallCheck tests properties for all the finitely many values up to some depth, progressively increasing the depth used. For data values, depth means depth of construction. For functional values, it is a measure combining the depth to which arguments may be evaluated and the depth of possible results.

The package is based on the paper by Colin Runciman, Matthew Naylor and Fredrik Lindblad.

smallcheck's Issues

Build on GHC 7.0.x

Any chance we can get this package to build on GHC 7.0.x? The Snap Framework commits to supporting the past few major Haskell Platform releases and for now that includes GHC 7.0. Right now our buildbot is getting a build failure from smallcheck.

Test/SmallCheck/Series.hs:29:14:
    Unsupported extension: DefaultSignatures

Add haddock docs for API

build errors with GHC-7.4

Building library for smallcheck-1.2.1..
[1 of 6] Compiling Test.SmallCheck.SeriesMonad ( Test/SmallCheck/SeriesMonad.hs, dist/build/Test/SmallCheck/SeriesMonad.o )

Test/SmallCheck/SeriesMonad.hs:41:31:
    No instance for (Monad m)
      arising from a use of `fmap'

EDIT: This looks like some logict version bound issue, so I'll let you figure it out.

Unify depthCheck and smallCheck

At the moment, there are two ways to run smallcheck: for depth bound 'd' only, or for depth bounds from 0 to 'd'.

The first option doesn't duplicate the work, while the second guarantees that we get the smallest counterexample.

Is it possible to have the best of both worlds?

Test Generic instances

E.g. test that the Generic instance matches a hand-written one.

See this report for an example of what can go wrong.

Better Char/String instances

See this reddit thread.

To do this, we need to decide:

• what kinds of characters we want to generate, and at which rate
• make newtype wrappers for some common cases? ASCII, Printable, Unicode?

questionable wording of conterexample output (there exists 0 1 such that...)

Hi.

The 1.0 release looks good, and contains some features that I sure want to use.

Nit-pick-ingly, I don't know whether the following change is an improvement:

smallcheck-0.6

smallCheck 3 $ \ x y -> x+y==x-(y :: Int)
Depth 0:
  Completed 1 test(s) without failure.
Depth 1:
  Failed test no. 1. Test values follow.
  -1
  -1

smallcheck-1.0.2

smallCheck 3 $ \ x y -> x+y==x-(y :: Int)
Failed test no. 2.
there exist 0 1 such that
  condition is false

The phrase "there exist 0 1 such that" is mathematically meaningless
(the numbers 0 and 1 do exist anyway), and actually a syntax error (after "exists", a variable must follow). I am sensitive to this because of using smallcheck in teaching.

It would be simpler, and mathematically correct, to just say something like the following?

condition 0 1 == False

Port some convenience list functions from QuickCheck

I have a function which takes a list as an argument, and I would like to test that its result does not depend on the order of the list (sort comes to mind as a realistic example). QuickCheck has a function to do this: shuffle.

I have thrown together a simple implementation for SmallCheck here.

How to generate a non-empty list of items with a predicate (using suchThat)?

I've copied suchThat: https://github.com/feuerbach/smallcheck/blob/8630db2a59aa942a4ac25fb42b6f957c1fb1fca3/Test/SmallCheck/Series.hs#L231

into my own codebase, since it isn't exported by SmallCheck, but is really useful.

I'm having some problems using it to generate a non-empty list, however, since I end up with really inefficient code:

import Test.SmallCheck.Series
import qualified Test.SmallCheck.Series as SS

instance Serial m (OrderBook venue base quote) where
   series = do
      midPrice <- series
      let buyOrderProp o  = oPrice (buyOrder o) < midPrice
          sellOrderProp o = oPrice (sellOrder o) > midPrice
      SS.NonEmpty buyOrders  <- series `suchThat` (all buyOrderProp . SS.getNonEmpty)
      SS.NonEmpty sellOrders <- series `suchThat` (all sellOrderProp . SS.getNonEmpty)
      return $ OrderBook (Vec.fromList $ sort buyOrders)
                         (Vec.fromList $ sort sellOrders)

When depth gets greater than 5, tests grind to a halt since any list which contains just a single item that doesn't fulfill the predicate is discarded.

Is there a way to use suchThat in combination with NonEmpty to generate a non-empty list of items that all adhere to a predicate on a per-item basis (as opposed to applying suchThat to the entire list)?

Side note: when using the above series implementation, SmallCheck fails to find counterexamples at depth 5, even though it successfully finds counterexamples at the same depth when using a more performant series implementation (which uses Test.SmallCheck.Series.list in combination with suchThat (although this doesn't work for non-empty lists)).

Clarify the deprecation statement

The README currently says:

As of 2023, this library is largely obsolete: arbitrary test generators with shrinking such as falsify offer much better user experience.

Can someone clarify this? For example, does falsify still exhaustively test data combinations up to a bound, or does this just mean that this approach is no longer considered viable?

Does anyone have an example of how falsify can replace smallcheck?

Integrate with test-framework

See http://hackage.haskell.org/package/test-framework

smallcheck-1.1.1 doesn't compile with current GHC head snapshot

Citing from http://hydra.cryp.to/build/794575/nixlog/13/raw:

Test/SmallCheck/Property.hs:187:10:
    Could not deduce (Monad n)
      arising from the superclasses of an instance declaration
    from the context: (Monad m, m ~ n)
      bound by the instance declaration
      at Test/SmallCheck/Property.hs:187:10-52
    Possible fix:
      add (Monad n) to the context of the instance declaration
    In the instance declaration for ‘Testable n (Property m)’

Smallcheck defines an orphan which overlaps `Show (a -> b)` instance from base

https://hackage.haskell.org/package/base-4.8.2.0/docs/Text-Show-Functions.html

/home/mgsloan/fpco/stack/src/Stack/Types/Nix.hs:33:13:
    Overlapping instances for Show
                                (Maybe Resolver -> Maybe CompilerVersion -> Text)
      arising from the sixth field of ‘NixOpts’
        (type ‘Maybe Resolver -> Maybe CompilerVersion -> Text’)
    Matching instances:
      instance [safe] Show (a -> b) -- Defined in ‘Text.Show.Functions’
      instance [overlap ok] (Test.SmallCheck.Series.Serial
                               Data.Functor.Identity.Identity a,
                             Show a, Show b) =>
                            Show (a -> b)
        -- Defined in ‘Test.SmallCheck.Series’
    When deriving the instance for (Show NixOpts)

Serial and CoSerial instances for `Word`

It'd be basically replacing the internal N wrapper with Word and export it. At a first glance it looks easy to replace N, wherever is used, with Word.

I can submit a PR myself, if you think this is alright.

With this change I could also include the instances for Word8, Word16, Word32, Word64, Int8, Int16...

Add 'tasty' instead of test-framework to docs as an option for running smallcheck

Hi,

You may already be aware of this, but the docs still suggest using test-framework:

https://github.com/feuerbach/smallcheck/blob/master/Test/SmallCheck.hs#L79

As you know, it's in a bad way these days, so this could lead unwary people in the wrong direction. It might be better to either replace the reference completely with a pointer to the tasty docs, or at least clarify that tasty is a (better :-) ) option.

depth for generics is broken

when using Generics, and incorrect depth is used:

> data Test = Test1 Int | Test2 String | Test3 String deriving (Generic, Show)
> instance Monad m => Serial m Test
> list 1 series :: [Test]
[Test1 0]
> list 2 series :: [Test]
[Test1 0,Test2 "",Test1 1,Test3 "",Test1 (-1)]

it looks like it's using the depth of the Rep Test, instead of the depth of Test.

Review `v1.1.1` restoration commit before pushing

@feuerbach I hand picked the revert commits and squashed them all into a single commit which is directly fast-forwardable to feuerbach/master. You can check the range from where I picked the commits here.

It builds and tests successfully. I think I'm not discarding any non-experimental commit but please have a look before I push it to the main repo.

I'll open a new issue about partial functions and arrows linking to the old commits in case we try to implement them in the future.

After solving this issue, I don't think I'll require that much attention, but this was a tricky revert and wouldn't like to screw it all up with my first commit :)

Typeable changes cause smallcheck not to build in HEAD

Configuring smallcheck-1.0.2...
Building smallcheck-1.0.2...
Preprocessing library smallcheck-1.0.2...
[1 of 6] Compiling Test.SmallCheck.SeriesMonad ( Test/SmallCheck/SeriesMonad.hs, dist/build/Test/SmallCheck/SeriesMonad.o )
[2 of 6] Compiling Test.SmallCheck.Series ( Test/SmallCheck/Series.hs, dist/build/Test/SmallCheck/Series.o )
[3 of 6] Compiling Test.SmallCheck.Property.Result ( Test/SmallCheck/Property/Result.hs, dist/build/Test/SmallCheck/Property/Result.o )
[4 of 6] Compiling Test.SmallCheck.Property ( Test/SmallCheck/Property.hs, dist/build/Test/SmallCheck/Property.o )

Test/SmallCheck/Property.hs:64:10:
    Not in scope: type constructor or class ‛Typeable1’
    Perhaps you meant ‛Typeable’ (imported from Data.Typeable)

Test/SmallCheck/Property.hs:66:5:
    ‛typeOf’ is not a (visible) method of class ‛Typeable’

Improve showing functions

First, provide a Fun datatype which knows about the function structure and show it instead.

Second, implement either function shrinking as described by Klaessen, or lazy-smallcheck-like shrinking.

Merge smallcheck and lazysmallcheck

See http://hackage.haskell.org/package/lazysmallcheck

Extract Number Wrappers

There is a lot of overlap with number newtypes like Negative, Positive with QuickCheck. It would be nice to share those types between the various testing packages.

Would you consider allowing an external package for these types?

How to define instances for bounded enum types?

In earlier versions of SmallCheck, it was quite simple to define a Serial instance for bounded enume types. In the new version, however, the same use case seems to have gotten much more complicated:

data Color = Black | Brown | Blue | Green
  deriving (Show, Read, Eq, Ord, Bounded, Enum)

instance Monad m => Serial m Color where
  series = foldr1 (\/) (map pure [minBound..maxBound])

Am I missing something? Is there a simpler solution to achive this?

test function of Property type class is not available

I would like to run (all) my tests using QuickCheck and SmallCheck using Hspec. I Thought it might be helpful to run both test cases using the same Spec structure as when running test cases with either of them.

My idea was to create a library which handles the importing of QuickCheck and SmallCheck (and maybe other testing libraries) and reexports an unique interface for running both. The implementation is very simple.
That is why I present the basic code here (including a Spec example):

{-# LANGUAGE TypeFamilies #-}
{-# LANGUAGE MultiParamTypeClasses #-}
{-# LANGUAGE FlexibleInstances #-}
{-# LANGUAGE FlexibleContexts #-}
module Data.ListSpec where

import qualified Test.QuickCheck       as QC
import qualified Test.SmallCheck       as SC
import qualified Test.Hspec.SmallCheck as SC
import           Test.Hspec            hiding (it)
import qualified Test.Hspec            as Hspec

spec :: Spec
spec = do
  describe "length" $
    it "is higher for longer lists" $ property $ \ xs ->
      not (null xs) ==> length xs > length (tail (xs :: [Int]))

it descr p = do
  Hspec.it (descr ++ " (SC)") $ SC.property p
  Hspec.it (descr ++ " (QC)") $ QC.property p

infixr 0 ==>
(==>) :: (SC.Testable m prop, QC.Testable prop)
  => Bool -> prop -> (SC.Property m, QC.Property)
cond ==> p = (cond SC.==> p, cond QC.==> p)

property :: (SC.Testable IO prop, QC.Testable prop)
  => prop -> (SC.Property IO, QC.Property)
property p = (SC.property p, QC.property p)

instance QC.Testable (SC.Property m, QC.Property) where
  property (_, p) = QC.property p

instance (Monad m, m ~ n) => SC.Testable n (SC.Property m, QC.Property) where
  test (p, _) = SC.test p

Unfortunately it is not possible to implement the last Testable instance for SmallCheck as function test is not being exported. Thus, my suggestion is to export it and rerelease SmallCheck. I understand the Test.SmallCheck.Property module (and its function test) is internal. If it is required to stress this, I would suggest renaming this module to Test.SmallCheck.Internal.Property, exporting the test function and making it a visible module of the package.

Of course I am open to suggestions to achieve my goal (without code duplication) otherwise.

Limit the number of tests

twanvl writes:

The default smallCheck could just have a limit on the number of testcases and report the depth searched:

λ> smallCheck prop_abc
Ok, stopped after 1000 tests, tested all values up to depth 3.

Show the bad tests(the ones which didn't meet the ==> condition) even on success

I often see this result from checking a property which has ==> somewhere in it :

Completed n tests without failure.
But w did not meet ==> condition.

Now when w==n, I would like to see some of the tests to know why wasn't the ==> met, so maybe print them ? or save them in a log if w is big.

Could not find module Numeric.Natural in GHC 7.8

smallcheck-1.1.3 fails to build with GHC 7.8 (https://api.travis-ci.org/v3/job/317785501/log.txt), because Numeric.Natural appeared in base only in GHC 7.10. I suggest either to bump minimal bound of base, or to add

if impl(ghc < 7.10)
  build-depends: nats

Generically derived Serial instance for mutually recursive data type causes infinite loop

Hi there,

Firstly, thanks for making this wonderful library!

When automagically deriving Serial instances for Generic data types that are recursive, smallcheck seems to go into an infinite loop when trying to generate values of this type.

Here is a source repository/commit that exhibits this symptom: runeksvendsen/rule-lang@b2f36ef

Steps to reproduce:

git clone https://github.com/runeksvendsen/rule-lang.git
cd rule-lang
git checkout b2f36ef6fb130a2e429de3d22d08f76b76c60a65
stack test

The above will hang indefinitely using 100% CPU without generating any values.

This is fixed by manually creating Serial instances that, for some type A, applies decDepth to any generator that ends up creating an A.

Documentation: types with >4 fields

http://stackoverflow.com/q/20581177/110081

Work with streams

Hi,

I would like to generate some streams that I'll fold over. I have:

data Event e = Event {
     _event_id        :: EventId
   , _event_timestamp :: UTCTime
   , _event           :: e
   } deriving (Show, Functor, Generic) -- Generic added to give me an easy instance of Serial

How would one go about generating a stream of these that have both event_id and event_timestamp increasing?

test-framework-smallcheck fails to build on GHC 7.4.1

See https://travis-ci.org/kowey/GenI/builds/4188943/#L442

Confusingly enough, I seem to have a working copy on my GHC 7.4.1, perhaps installed with cabal-dev…

Completed 0 tests without failure.

Using smallcheck-1.0.2, the following doesn't manage to come up with any examples.

  smallCheck 8 $ \f -> (f [True]::Bool)
  Completed 0 tests without failure.

I think that for this case, "Completed .. without failure" is not a good way of reporting this. Shouldn't the test count as failure if no testcases can be generated?

Smallcheck fails to build due to the Safe pragma

Smallcheck fails to build due to the Safe pragma. The logict dependency is the problem. Adding appropriate version bounds would likely resolve the issue.

You can see logs of an example failed compilation here.

Document that 'derive' library can derive Serial instances

See http://community.haskell.org/~ndm/darcs/derive/derive.htm

improve documentation of Serial method

at the moment it says

class Serial a where series :: Series a
type Series a = Depth -> [a]
-- Series is a function from the depth to a finite list of values. 
type Depth = Int
-- Maximum depth of generated test values 

this is not helpful when constructing instances:

the missing information is that in a Serial instance,
"series d" should give all values of depth smaller or equal to d.

otherwise, the consN combinators do not work correctly.

Generate the tests of depth exactly n?

http://www.reddit.com/r/haskell/comments/18uhfy/announcing_smallcheck_10/c8igxeo

Fix the tests!

Builf failure: Test.SmallCheck.Series: GHC.Base.MonadPlus ml

Good day Bodigrim, and to Odessa.

On the recent Hackage Cabal build downstream:
Live CI log: https://github.com/haskell-nix/hnix-store/runs/1712505541?check_suite_focus=true

Backup of the build log

Resolving dependencies...
Build profile: -w ghc-8.10.3 -O0
In order, the following will be built (use -v for more details):
 - logict-0.7.1.0 (lib) (requires download & build)
 - profunctors-5.6.1 (lib) (requires download & build)
 - smallcheck-1.2.0 (lib) (requires download & build)
 - saltine-0.1.1.1 (lib) (requires download & build)
 - tasty-smallcheck-0.8.2 (lib) (requires download & build)
 - hnix-store-core-0.4.0.0 (lib) (first run)
 - tasty-hspec-1.1.6 (lib) (requires download & build)
 - hnix-store-core-0.4.0.0 (test:format-tests) (first run)
Downloading  logict-0.7.1.0
Downloaded   logict-0.7.1.0
Downloading  smallcheck-1.2.0
Starting     logict-0.7.1.0 (lib)
Downloaded   smallcheck-1.2.0
Downloading  tasty-smallcheck-0.8.2
Building     logict-0.7.1.0 (lib)
Downloaded   tasty-smallcheck-0.8.2
Downloading  profunctors-5.6.1
Downloaded   profunctors-5.6.1
Downloading  saltine-0.1.1.1
Starting     profunctors-5.6.1 (lib)
Downloaded   saltine-0.1.1.1
Downloading  tasty-hspec-1.1.6
Building     profunctors-5.6.1 (lib)
Downloaded   tasty-hspec-1.1.6
Installing   logict-0.7.1.0 (lib)
Completed    logict-0.7.1.0 (lib)
Starting     smallcheck-1.2.0 (lib)
Building     smallcheck-1.2.0 (lib)
Installing   profunctors-5.6.1 (lib)
Completed    profunctors-5.6.1 (lib)

Failed to build smallcheck-1.2.0.
Build log (
/home/runner/.cabal/logs/ghc-8.10.3/smallcheck-1.2.0-d7480ac1ddd1f71fbb92777ab7e97596406f7a336703a64f170536e159f25499.log
):
Configuring library for smallcheck-1.2.0..
Preprocessing library for smallcheck-1.2.0..
Building library for smallcheck-1.2.0..
cabal: Failed to build smallcheck-1.2.0 (which is required by
test:format-tests from hnix-store-core-0.4.0.0). See the build log above for
details.

[1 of 6] Compiling Test.SmallCheck.Property.Result ( Test/SmallCheck/Property/Result.hs, dist/build/Test/SmallCheck/Property/Result.o, dist/build/Test/SmallCheck/Property/Result.dyn_o )
[2 of 6] Compiling Test.SmallCheck.SeriesMonad ( Test/SmallCheck/SeriesMonad.hs, dist/build/Test/SmallCheck/SeriesMonad.o, dist/build/Test/SmallCheck/SeriesMonad.dyn_o )
[3 of 6] Compiling Test.SmallCheck.Series ( Test/SmallCheck/Series.hs, dist/build/Test/SmallCheck/Series.o, dist/build/Test/SmallCheck/Series.dyn_o )

Test/SmallCheck/Series.hs:276:14: error:
    • Could not deduce (GHC.Base.MonadPlus ml)
        arising from a use of ‘mzero’
      from the context: Monad m
        bound by the type signature for:
                   limit :: forall (m :: * -> *) a.
                            Monad m =>
                            Int -> Series m a -> Series m a
        at Test/SmallCheck/Series.hs:272:1-64
      or from: MonadLogic ml
        bound by the type signature for:
                   go :: forall (ml :: * -> *) b. MonadLogic ml => Int -> ml b -> ml b
        at Test/SmallCheck/Series.hs:275:5-46
      Possible fix:
        add (GHC.Base.MonadPlus ml) to the context of
          the type signature for:
            go :: forall (ml :: * -> *) b. MonadLogic ml => Int -> ml b -> ml b
    • In the expression: mzero
      In an equation for ‘go’: go 0 _ = mzero
      In an equation for ‘limit’:
          limit n0 (Series s)
            = Series $ go n0 s
            where
                go :: MonadLogic ml => Int -> ml b -> ml b
                go 0 _ = mzero
                go n mb1
                  = do cons :: Maybe (b, ml b) <- msplit mb1
                       ....
    |
276 |     go 0 _ = mzero
    |              ^^^^^
Error: Process completed with exit code 1.

Changes like these can be caught by scheduling CI builds.

inconsistent naming/typing in exists/forall

"exists" and "forall" should have idential types?

exists :: (Show a, Serial a, Testable b) => (a -> b) -> Property

forAll :: (Show a, Testable b) => Series a -> (a -> b) -> Property

(I'd prefer to have a version of "forall" without the Series a argument.
Yes, I know I can just test the function, but writing explicit "forall" looks
much better.)

Serial instance for base types

Having only default Prelude instances is kind of a deal-breaker.
Prelude is in base, so nothing should speak against providing instances for all of its types.

non-IO test driver (for multiple tests)

I want to write code like this:

append :: [a] -> [a] -> [a]
append a b = case a of
  [] -> b
  x : y -> x : append y b

assoc op = \ a b c ->
    op a (op b c) == op (op a b) c

comm op = \ a b -> op a b == op b a

left_neutral op a = \ b -> op a b == b 
right_neutral op b = \ a -> op a b == a 

tests :: Either String String
tests = do
    testE "assoc" 1000 
        $ assoc ( append ::[Bool] ->[Bool]->[Bool])
    testE "right_neutral" 1000 
        $ right_neutral ( append ::[Bool] ->[Bool]->[Bool]) []
    testE "left_neutral" 1000 
        $ left_neutral ( append ::[Bool] ->[Bool]->[Bool]) []

it should have

• a pure test driver (no IO) so I can process the result in a pure program and without re-parsing
• run several tests, give OK only if all pass
• give human-readable output

I can achieve this with

import Test.SmallCheck.Property
import Control.Monad.Error

testE :: Testable a 
      => String -> Int -> a
      -> Either String String
testE name num prop 
  = case 
        filter ( \ r -> not (resultIsOk $ result r))
      $ take num
      $ do depth <- [0..] ; test prop depth
    of [] -> Right $ unwords 
          [ "property", name, "passed", show num , "tests" ]
       r : _ -> Left $ unwords $
         [ "property", name, "fails for inputs"
         ] ++ arguments r

so I guess I'd like to have that (or something similar to the same effect) in the library somewhere.