jcornaz / kwik

Property-based testing library for Kotlin

Kwik

Property-based testing library for Kotlin.

Main features:

• Test-engine agnostic
• Multiplatform
• No reflection
• Configurable built-in generators
• Easy way to create and combine generators
• Seeded generation for reproducible results

Find more information on https://kwik.readthedocs.io

Status

This project is discontinued. For property based testing with Kotlin, check out kotest.

How it looks like

class PlusOperatorTest {

    @Test
    fun isCommutative() = forAll { x: Int, y: Int ->
        x + y == y + x
    }

    @Test
    fun isAssociative() = forAll(iterations = 1000) { x: Int, y: Int, z: Int ->
        (x + y) + z == x + (y + z)
    }

    @Test
    fun zeroIsNeutral() = forAll(seed = -4567) { x: Int ->
        x + 0 == x
    }
}

For more information read the usage and look at the available generators

Motivation

Property based testing is great and very powerful. But despite the fact that many good libraries already exist, none of them fully fit my needs.

The known alternatives either:

• Are bound to a specific test-engine
• Can only be used when compiling kotlin to Java (and cannot be used in multi-platform projects)
• Relies on reflection, making the tests slower and make some errors detectable only at runtime
• Do not allow enough freedom and safety to customize existing generators
• Force the user to add unwanted dependencies in the classpath

Requirements

• Kotlin version: 1.4.0 or newer
• JDK version: 8, 11 or 15

Setup

Example of setup using gradle.

repositories {
    jcenter()
}

dependencies {
    testCompile("com.github.jcornaz.kwik:kwik-core-jvm:$kwikVersion")
}

Find more detailed information in the setup instructions.

Contribute

See how to contribute

kwik's Issues

Change `ratio` for `probability` in `withSamples` and `withNull`

Currently, the samples are emitted as soon as the requested ratio is reached. This may lead to miss interesting inputs when generators are combined.

It would be better to trade the ratio for a probability and randomize whenever the values are emitted.

Provide a merge operator

The following should be possible (and also provided):

• nonZeroInts() = ints(max = -1) + ints(min = 1)
• nonZeroDoubles() = doubles(max = -Double.MIN_VALUE) + ints(min = Double.MIN_VALUE)

Reintroduce checkForAll

checkForAll was an alternative to forAll which requires to call assertions instead of returning a boolean.

It has been removed in #29, but removing it was not a good Idea.

Yes checkForAll is a little bit less safe than forAll because it allows writing code that compiles and which does not actually check the property (if the user forget to write assertions)

But it also has advantages: Using assertions instead of boolean expressions allow to print a more detailed falsification report explaining what was the value obtained vs the expected value.

So at the end choosing between forAll and checkForAll it is a trade-off decision that is up to the user of this library.

And checkForAll is anyway still as safe as any other test method in most other test frameworks.

Document runner mechanic

Travis setup

A combining generator should start by the exaustive combination of the three first emitted values.

By conventions and by the behavior of withSamples and withNull, the generatos always start by emitting their edge-cases, starting from the most important to the least one.

For example Generator.ints() will always start by the following values: 0, 1, -1, Int.MAX_VALUE, Int.MIN_VALUE.

These values are considered to have the highest chance to prove the property is wrong (and falsify it), which is why we start by testing them before testing random values.

When we combine two generators it would then returned generator combine the first values of the combined generator as well.

For instance, combining Generator.Int() with itself would start by returning the following pairs: 0,0, 0,1, 0,-1, 1,0, 1,1, 1,-1, -1,0, -1,1, -1,-1

String generator

Document generators

Add ability to combine more than 2 operators

Here is how it could look like:

val generator = Generator.combine(
  Generator.uuids(),
  Generator.strings(),
  Generator.strings(),
  Generator.naturalInts()
) { (id, firstName, lastName, age) ->
  Person(id, firstName, lastName, age)
}

Rename generator combine function

I think zip is not clear and does not fit the case of random sequences. I'd rather simply call it combine.

Primitive generators

Random.nextString()

Generator with multiple inputs

Initial support up to 4 random inputs seems reasonable to me

Make primitive generators support shrinking

Code quality analysis

Without test code coverage: No good support for Kotlin multiplatform yet.

Sequence generator

Allow to customize the default number of iteration via system property

One should be able to run less or more iteration depending on the environment.

Example of use cases:

• on a weekly build I may want to massively increase the number of iteration because test time is not a problem there and I want to maximize the number of teste cases.
• on a fresh local clone already verified by the CI, running just a few (or 1) iteration is enough to make sure my local clone is in good state, and I want the test to run as fast as possible.
• Before opening a pull request I may want to run some tests with a huge number of iteration just once.

Usage example:

gradlew test -Dkwik.iterations=100000 # on CI
gradlew test -Dkwik.iterations=10 # locally

Publish sources and documentation artifacts

Remove checkForAll

It is not a safe api, since it allows the user to forget to put assertions, in which case the test will pass for wrong reasons.

Use code generation for generating overloads of `forAll` and equivalents

It is tedious to test and maintain all overloads of forAll. And the same kind of problem will arise again with #77 (combine more than 2 operators) and #8 (spek integration).

It would be nice to leverage code generation (maybe with Kotlin poet?) to easily and safely support more number arguments for these functions.

In case of a falsified property, display how many test have been executed.

It helps to understand if the property is "never true" or if it is false only in "certain cases".

Spek integration

For specification style:

object ExampleSpec : Spek({
  describe("plus operator") {
    property("is associative") { x, y -> x + y == y + x }
  }
})

And for gherkin style:

object ExampleSpec : Spek({
  Feature("plus operator") {
    property("is associative") { x, y -> x + y == y + x }
  }
})

Note: It would be in a dedicated gradle module. Kwik's users should to opt-in for the Spek extensions. The module name could be something like kwik-spek-jvm (Due to limitations on Spek's side, it cannot be multiplatform at the moment.)

Distinct operator

Here is how the API could look like:

/**
 * Returns a generator that emits items of the upstream generator only once (based on result of `hashCode` and `equals` method)
 *
 * Be sure to not overuse it has it may slow down your tests
 */
fun <T> Generator<T>.distinct(): Generator<T> = TODO()

Make collection generators support shrinking

Document how to contribute

• Discussion places (issues & gitter)
• Clarify when to create a pull request against master or develop
• Explain how to work with the sources
  • Build from sources
  • Run the tests
• Explain how to contribute to the documentation
  • How to build the documentation locally

DateTime Generators

It would be nice to have generators for the most common types of java.time api (instant, zone, localDate, localTime, localDateTime, etc.)

It would obviously be available only for the JVM.

Random.nextList()

And equivalents for sets and maps

Collection generators

• List
• Set
• Map

UUID generator

Shrinking

This is "epic" that regroup multiple issues. The actionable sub-tasks are tracked on zenhub: https://app.zenhub.com/workspaces/kwik-5e3aa5f35d8a250b41d730e1/issues/jcornaz/kwik/5

Write examples

Make generator operators keep the shrinkability of the source generator

Allow to skip evaluation

Example:

@Test
fun md5ReturnsDifferentHashForDifferentInput() {
  forAll { str1: String, str2: String ->
    skipIf(str1 == str2) // <-- Need to find a good name/api for it

    md5(str1) != md5(str2)
  }
}

Enum generator

Add an `Arbitrary` typealias for `Generator`

Despite Generator being quite natural for a Java developer, I think Arbitrary have some good advantages:

• It is consistent with the naming scheme in the Haskell QuickCheck library.
• It nicely creates English sentences: forAll(Arbitrary.int()) { x -> ... } = "for all arbitary int x, ..."

Provide few common charsets for the string generators

Should be provided:

• All characters
• Printable characters
• AlphaNum
• Alpha

And some operators like:

• LowerCase
• UpperCase

Build JavaScript artifacts

Make the falsified message more verbose and clear about the arguments

It should be something like that:

Property falsified after 42 attempts (out of 200)
Argument 1: 3
Argument 2: test
Argument 3: true
Generation seed: 123

Since it would contain all useful information, the other information printed (#17) should be printed only in case of success. (And could be more succinct as well)

Add a filter operator for Generator

Despite such operator should be avoided for performance reasons, there are cases where it is a necessity.

Print informations

The user should at least know:

• what was the seed used (Allowing to reproduce locally a result observed on CI for instance)
• how many tries have been performed (especially in case of failure so that it gives a feeling about if the property is never verified or if it is only falsified in some rare cases)

Extract generator API module

A generator collection library (or module) should not need to depend on core.

Such a library/module only needs:

• The generator interface
• Functions an utilities for generator creation and composition

Some may also want the build-in operator to reuse them. (many operators can be built as a combination of the primitive operators)

Combine generators

Publish to bintray

Make string generator support shrinking

The default max length of collections and string should always be >= the min length

Prints more relevant information in case of exception

Currently, if the code being tested throws an exception, there is no clue about what were the generated arguments.

Kwik should catch such exception and wrap them to give the usual information (number of attempts/iterations, arguments, etc.)

Make possible to require that a predicate is satisfied for at least one generated input set

Here is how it could look like:

forAll { x: Int ->
  ensureAtLeastOne { x > 10 }

  // ...
}

And the property would run for at least the number of iterations (200 by default) and as long as necessary to satisfy at least once the lambda { x > 10 }

Flow generator

At first it can be a simple map of the sequence generator doing asFlow.

More advanced configuration (delay, infinite flow) can be implemented later when requested.

Default generators

Should try to infer what generator to use for a given type

Provide aliases for some common generators configuration

• positiveInts() = ints(min = 0)
• naturalInts() = ints(min = 1)
• negativeInts() = ints(max = -1)
• positiveDoubles() = doubles(min = 0)
• negativeDoubles() = doubles(max = -1)
• nonEmptyStrings() = strings(minLength = 1)
• nonEmptyLists() = lists(minSize = 1)