szktty / kiri-check

A testing library for property-based testing in Dart, enabling custom test data generation, testing stateful systems, and integrating seamlessly with package:test.

kiri-check

kiri-check is a property-based testing library for Dart.

Features

• Integrated with package:test, it can be used in the same way as regular tests. Additionally, property-based tests can be added without modifying existing test code.
• Customization of the exploration method is easy. You can specify data ranges, increase or decrease the number of trials, and prioritize edge cases.
• Supports stateful testing. You can test the behavior of a system that changes state over time.

Install

Install the library from pub.dev using the following command:

With Dart:

dart pub add dev:kiri_check

With Flutter:

flutter pub add dev:kiri_check

Alternatively, add the library to your pubspec.yaml and run dart pub get or flutter pub get.

dev_dependencies:
  kiri_check: ^1.1.0

Documentation

Please refer to the Documentation.

Basic usage

Properties can be implemented as tests using package:test. Assertions use functions from package:test.

1. Import the kiri-check library.

   import 'package:kiri_check/kiri_check.dart';

2. Implement properties using the property function. This function takes the title of the test and the function to execute the test as arguments.
3. Implement the test to validate test data using the forAll function. This function takes an arbitrary that generates random test data as an argument.

Example:

import 'package:kiri_check/kiri_check.dart';

dynamic fizzbuzz(int n) {
  // Implement me!
}

void main() {
  property('FizzBuzz', () {
    forAll(
      integer(min: 0, max: 100),
          (n) {
        final result = fizzbuzz(n);
        if (n % 15 == 0) {
          expect(result, 'FizzBuzz');
        } else if (n % 3 == 0) {
          expect(result, 'Fizz');
        } else if (n % 5 == 0) {
          expect(result, 'Buzz');
        } else {
          expect(result, n.toString());
        }
      },
    );
  });
}

Stateful testing

Stateful testing allows you to test the behavior of a system that changes state over time. This is particularly useful for testing stateful systems like databases, user interfaces, or any system with a complex state machine.

To perform stateful testing, in addition to importing kiri_check/kiri_check.dart, you need to import kiri_check/stateful_test.dart.

Example:

import 'package:kiri_check/kiri_check.dart';
import 'package:kiri_check/stateful_test.dart';

// Abstract model representing accurate specifications
// with concise implementation.
final class CounterModel {
  int count = 0;

  void reset() {
    count = 0;
  }

  void increment() {
    count++;
  }

  void decrement() {
    count--;
  }
}

// Real system compared with the behavior of the model.
final class CounterSystem {
  // Assume that it is operated in JSON object.
  Map<String, int> data = {'count': 0};

  int get count => data['count']!;

  set count(int value) {
    data['count'] = value;
  }

  void reset() {
    data['count'] = 0;
  }

  void increment() {
    data['count'] = data['count']! + 1;
  }

  void decrement() {
    data['count'] = data['count']! - 1;
  }
}

// Definition of stateful test content.
final class CounterBehavior extends Behavior<CounterModel, CounterSystem> {
  @override
  CounterModel initialState() {
    return CounterModel();
  }

  @override
  CounterSystem createSystem(CounterModel s) {
    return CounterSystem();
  }

  @override
  List<Command<CounterModel, CounterSystem>> generateCommands(CounterModel s) {
    return [
      Action0(
        'reset',
        nextState: (s) => s.reset(),
        run: (system) {
          system.reset();
          return system.count;
        },
        postcondition: (s, count) => count == 0,
      ),
      Action0(
        'increment',
        nextState: (s) => s.increment(),
        run: (system) {
          system.increment();
          return system.count;
        },
        postcondition: (s, count) => s.count + 1 == count,
      ),
      Action0(
        'decrement',
        nextState: (s) => s.decrement(),
        run: (system) {
          system.decrement();
          return system.count;
        },
        postcondition: (s, count) => s.count - 1 == count,
      ),
    ];
  }

  @override
  void destroySystem(CounterSystem system) {}
}

void main() {
  property('counter', () {
    // Run a stateful test.
    runBehavior(CounterBehavior());
  });
}

For more detailed information on stateful testing, including advanced usage and customization options, please refer to Stateful Testing.

Known issues

• #13 Test results in IntelliJ IDEA do not reflect tearDownAll

TODO

• Example database
• Replace the PRNG with xorshift to improve performance and remove dependency on current PRNG
• Reimplement the cache mechanism with a reproducible PRNG using internal state

Author

SUZUKI Tetsuya ([email protected])

License

Apache License, Version 2.0

kiri-check's Issues

Version conflicts with Flutter dependencies during `flutter pub get`

Issue Description

When Flutter is listed as a dependency, running flutter pub get results in version conflicts for several packages.

Conflicting Packages

• meta
• collection
• test_api

Expected Behavior

flutter pub get should complete without version conflict errors.

Current Workaround

Currently, we're using dependency_overrides to manually specify versions for the conflicting packages:

dependency_overrides:
  collection: ^1.18.0
  meta: ^1.15.0
  test_api: ^0.7.0

Proposed Solution

Downgrade certain dependencies to align with Flutter's versions.

Steps to Reproduce

1. Add Flutter as a dependency in the pubspec.yaml file.
2. Run flutter pub get.
3. Observe version conflict errors for the mentioned packages.

Environment

• macOS Sonoma 14.5
• Dart 3.4.0
• Flutter 3.22.0 stable

Async support for stateless test callbacks

Overview

This issue addresses the implementation of asynchronous support for callback functions in stateless tests. Currently, only the test block of property supports asynchronous processing, while other callbacks do not.

Current problems

At present, callbacks such as setUp, tearDown, and the blocks taken as arguments by forAll do not support asynchronous processing. As a result, developers need to implement asynchronous operations using methods other than async/await in these callbacks, which increases the development burden.

Target APIs

• setUp, tearDown, and related functions
• Blocks taken as arguments by forAll

Proposed solution

We will modify the callback types using FutureOr. This change will allow support for asynchronous processing while maintaining compatibility with existing code.

Impact

This change is considered a breaking change. However, due to the use of FutureOr, existing code will not need to be modified.

Note

Asynchronous support for stateful tests will be addressed in a separate issue #10 .

Implement an arbitrary that accepts callable objects

Implement an arbitrary that accepts callable objects for dynamically generating values.
The arbitrary will enhance the flexibility and usability.

Proposed feature

• An arbitrary that takes a callable object as an argument and returns the result of evaluating that object as the generated value
• The callable object is newly evaluated each time a generation occurs
• Optional use of a Deck object

Implementation proposal

Arbitrary<T> build<T>(T Function(Deck) builder);

• The use of Deck is optional. It's fine not to use it.
• Shrinking is performed only on values generated by Deck. If the return value is not generated by Deck, it will not be shrunk.

Examples

Using Deck:

forAll(
  build((deck) => f(deck.draw(integer()))),
  (value) {
    // Test logic
  },
);

Not using Deck:

forAll(
  build((deck) => f()),
  (value) {
    // Test logic
  },
);

Async support for stateful testing

Description

Add support for asynchronous operations (async/await) in stateful testing to accommodate a wider range of use cases.

Current issue

The current stateful testing API doesn't support asynchronous operations. This makes it difficult to test systems involving async processes without resorting to non-standard synchronization methods.

Proposed solution

Applying FutureOr to relevant callbacks, allowing for async declarations. Target callbacks and functions:

• Methods in the Behavior class
• Methods in the Command interface
• Methods in the Action class

Support custom command classes in stateful testing

Purpose

Enable users to implement custom command classes for stateful testing.

Current limitation

Users are currently restricted to using Action commands to define process content in stateful tests.

Proposed change

Introduce support for user-defined Command subclasses or provide a base class for custom commands.

Benefit

Improved structure and reusability of complex command logic in stateful tests.

Add setup and teardown callbacks for all `forAll` loops

Problem

Currently, when users want to perform the same initialization and cleanup operations for all forAll loops, they need to write these operations individually for each forAll call. This leads to code duplication and makes it harder to maintain consistent setup and teardown logic across multiple property-based tests.

Proposed solution

Implement two new functions to set callbacks that will be executed before and after all forAll loops:

1. setUpForAll: A global function to set a callback that will be called before each forAll loop starts.
2. tearDownForAll: A global function to set a callback that will be called after each forAll loop ends.

Enable random value generation using arbitraries outside of tests

Description

We propose implementing a feature that allows random value generation using arbitraries outside of tests. Specifically, we suggest adding an example() method to each arbitrary.

Background and purpose

There is a need for a feature to generate random values using arbitraries to prepare test data in advance, outside of tests.
This feature will enable the preparation of test data that includes boundary values and special cases.

Proposed features

1. Add an example() method to each arbitrary class
2. Allow the generation of arbitrary test data before test execution using this method
3. Enable the specification of seed values in the example() method to reproduce the same values when necessary

Implementation Example

// Usage example
void main() {
  final intGen = integer(min: 0, max: 100);
  final stringGen = string(minLength: 1, maxLength: 10);

  // Generate data before the test
  final testData = [
    intGen.example(),
    stringGen.example(),
    intGen.example(),
  ];

  test('Test using pre-generated data', () {
    for (final data in testData) {
      // Test logic using the data
    }
  });
}

Differences from forAll

1. Usage context:

   • forAll: Used within tests to run property-based tests
   • example(): Used outside of tests to create test data

2. Shrinking feature:

   • forAll: Generated data is shrunk if the test fails
   • example(): Data is not shrunk as it is used outside of tests

3. Purpose:

   • forAll: Automatically generates and tests multiple random cases
   • example(): Manually generates data for specific test cases

Test results in IntelliJ IDEA do not reflect `tearDownAll`

Description

When running test code containing properties and tearDownAll in IntelliJ IDEA, the test results do not reflect the execution of tearDownAll. Even if an error occurs in tearDownAll, the test is shown as successful, and using print statements does not output to the tool window console. However, although tearDownAll appears not to be implemented, it is actually being executed.

This issue does not occur in VSCode. It is unclear whether this is a bug in kiri-check.

Environment

• macOS Sonoma 14.5
• Flutter 3.22.0
• Dart 3.4.0
• kiri-check 1.1.0
• IntelliJ IDEA 2024.1.4
• Dart plugin 242.19890.14

Steps to reproduce

Define tearDownAll and at least one property, then run the test from the IntelliJ IDEA run menu.

Here's a sample code:

import 'package:kiri_check/kiri_check.dart';
import 'package:test/test.dart';

void main() {
  tearDownAll(() {
    fail('tearDownAll failed');
  });
  property('dummy', () {});
}

The Dart plugin for IntelliJ IDEA uses the following command by default to run tests:

dart --no-serve-devtools run test -r json FILE

When this command is run directly from the terminal, you can confirm that the output JSON contains errors.

Workarounds

1. Run tests directly from the terminal
2. Use VSCode