graue / gentest Goto Github PK

Hey,
Could you put some info on how to run genTest?
thanks

Hello,
imagin:
var person = gentest.types.shape({
name: gentest.types.string,
age: gentest.types.int
});

now say we want to make sure age is always positive so we write a property like:

function chkPresonAnge() {
return person.age > 0;

}
now we want check this property. HOW?

Thanks;

This fails when run from a checkout of gentest:

> gentest.sample(gentest.types.number.nonZero)
GentestError: suchThat: could not find a suitable value
    at repl:1:10
    at REPLServer.self.eval (repl.js:110:21)
    at Interface.<anonymous> (repl.js:239:12)
    at Interface.emit (events.js:95:17)
    at Interface._onLine (readline.js:202:10)
    at Interface._line (readline.js:531:8)

Gentest has shrinking logic but it's not currently invoked by the test runner.

Could gentest generate a random binary tree or binary search tree? by random I mean a random depth with random values.

Using gentest to test itself seems dodgy, so I'm testing it with Mocha right now with explicit calls to gentest.sample and assertions about the generated values. I think this is fine for what it is, but much more should be tested, including a few higher-order generators that have been left out:

• test types.fmap
• test types.bind
• test types.suchThat
• test shrinking of each type
• test Property
• test Runner
• add something to measure test coverage

Also, it might be neat to test gentest by writing properties known to be true about the JavaScript standard library, and properties known to be false for specific inputs (making sure it finds those inputs).

Very interesting project, and I am considering it.
What I was wondering about is, how do you make tests repeatable?

it sounds like, if running a test twice it could fail once and pass once. That could cause issues.

Currently, Gentest only has a Node.js-based test runner. There should be a bundle you can stick into a script tag to run tests in browsers.

To make specific test runs reproducible, the gentest CLI should print the PRNG seed as part of its output, and allow setting the seed explicitly.

Perhaps npm publish this project?

Motivation: the invisible size parameter, which ramps up from 1 to 50 (by default, controllable with --max-size) over the course of a test run, controls both array lengths and the magnitude of generated integers. This means it is not possible to, for example, generate short arrays containing large numbers (e.g. 10 integers that are all in the millions), or long arrays containing small numbers (e.g., 100,000 integers between 0 and 10).

To enable this, we need to allow something like:

t.arrayOf(t.scaleSize(5, t.int.nonNegative))

In which scaleSize(mult, gen) would delegate to gen but with the size parameter multiplied by mult.

Reading the size is useful for creating recursive structures, so we should have that too. See "A generator for bounded trees" in this QuickCheck example.

Other implementations (test.check) also have combinators to set the size to a constant amount (not a multiplier), but I am less clear on why this would be useful.

In lib/Property.js, there's an interface that could allow people to write their own test runners and use Gentest solely as a library. It implements all the interesting stuff in Gentest in a purely functional way: defining properties with the forAll sugar, generating test cases for a property, running a test case against a property, and shrinking a failing test case (via an iterator, so the process can be stopped or paused at any step).

All that needs to be done is to:

• Expose this API publicly.
• Provide a way to go from the opaque "test case" type returned by genTest and consumed by runTest (which is, secretly, a tree) to a POJO, for display/logging.
• Ideally, document this API somewhere other than in the source.

(Among the benefits: This could provide a basis for adapters to use Gentest within Jasmine/Mocha/QUnit test suites.)

I think this project is shaping up well.
One question: tuples are very similar shape! I think one could imaging you bind them together e.g. tuples recursively generate new field. Right now it seems desecrate and binding these two save up more type.
One more question: right now numbers are not generator, what if I always want for a give variable get 5?

Gentest has a hard time with this property, converted from a ClojureScript double-check example by @jamii, and run with 1000 tests (-n 1000):

forAll([t.arrayOf(t.tuple([t.string, t.string, t.string]))],
       'simulate sparse/complex failure',
       function(_) {
         return Math.random() < 0.999;
       });

If all tests pass, it's fine. But sometimes (maybe 2 in every 10 test runs), Gentest tries hundreds of thousands of shrinks before running out of memory and crashing. This variant, where the inner array can be of any length instead of always 3, is even worse:

forAll([t.arrayOf(t.arrayOf(t.string))],
       'simulate sparse/complex failure',
       function(_) {
         return Math.random() < 0.999;
       });

Example output:

$ time gentest -n 1000 gtboom2.js 
Testing gtboom2.js
736/1000 simulate sparse/complex failure, shrinking 224/232359FATAL ERROR: CALL_AND_
RETRY_2 Allocation failed - process out of memory
Aborted (core dumped)

real    1m21.912s
user    1m0.212s
sys     0m2.467s

Perhaps what we need is a limit on either the total number of shrinks tried, the branching factor of the shrink tree, or both.

Could you provide more concrete examples on feature, probably some challenging cases that gentest could catch?

This could be as simple as saying, "Run gentest --help for a full list of options" in the readme. Worth having at least that.