dakusui / scriptiveunit

Verba volant, scripta manent.

ScriptiveUnit

ScriptiveUnit is a generic framework to build a JSON based DSTL (domain specific testing language) for mainly input-output systems such as search engines.

Installation

Include following xml fragment in your pom.xml.

    <dependency>
      <groupId>com.github.dakusui</groupId>
      <artifactId>scriptiveunit</artifactId>
      <version>{ScriptiveUnit Version}</version>
    </dependency>

For the released ScriptiveUnit versions, you can refer to this page.

Usage

Following is a diagram that illustrates how engineers and assets are interacting each other in an ecosystem where ScriptiveUnit is utilized.

Scripted test suite example

Full version of this example is found here.

    {
        "description":"An example test suite to Query-Result model on ScriptiveUnit",
        "factorSpace": {
            "factors": {
                "terms": [["ヒーター"], ["iPhone"]],
                "sortBy": ["price", "lexical", "relevance"],
                "order" : ["asc", "desc"]
            },
            "constraints": [
                ["if_then", ["equals", ["attr", "sortBy"], "relevance"], ["equals", ["attr", "order"], "desc"]]
            ]
        },
        "testOracles": [
            {
                "description": "Searching 'iPhone' should not return too many accessories",
                "given": ["equals", ["attr", "terms"], ["quote", "iPhone"]],
                "when": ["issue", ["query"] ],
                "then": [">",
                    ["size", ["filter", ["result"], ["containedBy", ["issue", ["with", {"terms":["iPhone&&シルバー"]}, ["query"]]]]]],
                    3
                ]
            },
            {
                "description": "Searching 'ヒーター' should also return items that contain 'ヒータ' or 'ストーブ'",
                "given": ["equals", ["attr", "terms"], ["quote", "ヒーター"]],
                "when": ["issue", ["query"] ],
                "then": ["<",
                  ["-",
                    ["size", ["issue", ["with", { "terms": ["ヒータ", "ストーブ"], "hits":-1 }, ["query"]]] ],
                    ["size", ["result"] ]
                  ],
                  2
                ]
            },
            {
                "description": "Valid queries should result in 200",
                "given": ["always"],
                "when": ["issue", ["query"]],
                "then": ["==", ["value", "statusCode", ["result"]], 200]
            }
        ]
    }

Script driver example

Full version of this example is found here.

    @Load(with = JsonBasedTestSuiteLoader.Factory.class)
    @RunWith(ScriptiveUnit.class)
    public class Qapi {
      ...

      @Import
      public Object collections = new Collections();

      @Import({
          @Alias(value = "*"),
          @Alias(value = "request", as = "query"),
          @Alias(value = "response", as = "result"),
          @Alias(value = "service", as = "issue")
      })
      public QueryApi<Request, Response, Entry> queryApi = new QueryApi<Request, Response, Entry>() {
        @Override
        protected Request buildRequest(Map<String, Object> fixture) {
          return new Request(fixture);
        }

        @Override
        protected Response service(Request request) {
          ...
          return new Response(matched);
        }

        @Override
        protected Request override(Map<String, Object> values, Request request) {
          ...
          return buildRequest(work);
        }
      };

      public static class Request {
        public static class Term {
          ...
        }
        ...
      }

      public static class Response extends LinkedList<Entry> implements Iterable<Entry> {
        ...
      }
    }

Output example

The Script and driver mentioned above will generate test results like following.

Future works

• "Regex" factor support
• Support 'preprocessing'
• Evaluation inside a JSON object
• Dependency mechanism

References

• JCUnit
• ActionUnit

scriptiveunit's Issues

Small refactorings

A place holder ticket for trivial refactorings.

Integrate "QueryAPI" support as a "first-class" behaviour mode

Rearchitecture scriptiveunit

Rearchitecture scriptiveunit to allow reuse its sub-components such as

• JSON (and other tree data structure) inheritance mechanism
• Lambda execution

Improve documentation

Implement a (programmable) direct execution mode

Bump up JCUnit version to 0.8.1

Bump up JCUnit version to 0.8.1.

Implement 'SuiteSet' feature

Implement 'SuiteSet' feature that runs all the scripts whose resource names matching given regex. Following is an example to illustrate how to use it.

@RunWith(ScriptiveSuiteSet.class)
@SuiteScripts(
    driverClass = Qapi.class,
    includes = { ".*api.json", ".*issue.*json", ".*print_twice.json" },
    excludes = { ".*iss.*" })
public class SuiteSetExample {
}

Update memoization design

Right now scriptive unit memoizes computation results of functions for each test oracle, that is belonging to a test case, but actually it should do it for each test case. It is because inputs of functions are coming from values in a test case and as long as a function to be memoized is properly designed, those inputs and outputs of the function will be kept the same during executing test oracles that belong to the test case.

Upgrade JCUnit

Upgrade JCUnit on which ScriptiveUnit depends to the latest version: 0.8.17

Support defun mechanism

, where if you define a function print_twice by

{
    "define": {
	"print_twice": ["sequential",
		     ["print", [0]],
		     ["print", [0]]
		    ]
	}
    }
}

, you can call it by

["print_twice", "Hello, world"]

, giving

Hello, world
Hello, world

Make test item names more descriptive

Implement a feature where a following template string used in description attribute of a test oracle,

        {
            "description": "{{@TESTSUITE}}: Searching {{_tems}} should also return items that contain 'ヒータ' or 'ストーブ'",
            "given": ["equals", ["attr", "terms"], ["quote", "ヒーター"]],
            "when": ["issue", ["query"] ],
...
        },

generates a following string displayed as a test item name of JUnit.

   001: Query-Result model example: Searching [ヒーター] should not return too many accessories

Implement 'tearDown' and 'tearDownAfterAll' mechanisms

Implement 'tearDown' and 'tearDownAfterAll' mechanisms.

Upgrade actionunit to 5.2.x

Improve output of ScriptiveUnit on test failures

Improve output of ScriptiveUnit on test failures.

Make it possible to run small amount of tests

In case we have thousands of test cases, the output/resource consumption of scriptive unit can become huge.
In this situation, we want to break down one scriptiveunit execution into pieces and run them parallel.

ScriptUnit cannot handle dividing non-terminating floating number

Caused by: java.lang.ArithmeticException: Non-terminating decimal expansion; no exact representable decimal result.
at java.math.BigDecimal.divide(BigDecimal.java:1690)
at com.github.dakusui.scriptunit.drivers.Arith.calc(Arith.java:52)
at com.github.dakusui.scriptunit.drivers.Arith.lambda$div$7(Arith.java:40)
... 127 more

Implement a statement execution mode

Make preprocessing part independent of the rest of the product

Loading and preprocessing mechanism of the scriptiveunit is useful not only for itself but also for other purposes than testing.
It is benefial to make it reusable.

Create a preprocessor mechanism

Implement a mechanism where you can preprocess resource files read by JsonBasedLoader by overriding JsonBasedLoader class's behaviour.

@Load(
    scriptPackagePrefix = "tests", scriptNamePattern = ".*\\.json", with = Qapi.Loader.class)
@RunWith(ScriptiveUnit.class)
public class Qapi {
  public static class Loader extends JsonBasedLoader {
    public Loader(Config config) {
      super(config);
    }

    @Override
    protected List<Preprocessor> getPreprocessors() {
      return singletonList(new Preprocessor() {
        ...
      });
    }
  }

Make it possible to reuse memoized result inside a test case

Currently memoized result can only be reused inside one test oracle and not be used another even if it belongs to the same test case.
By providing a way to reuse a computation result from another test oracle, we can make test executions more efficient.

Implement a callback mechanism

Provide programmable interface for Python

Use jpype, not jython.

Generate package-info.java from package-info.adoc automatically

Publish JavaDoc as a part of site documentation

Add unit tests

Followings are area where we should improve coverage.

• Implement tests for "extra" libraries

Lambda feature is missing

I want to be able to do following in testOracles.

            {
                "description": "Searching 'iPhone' should not return too many accessories",
                "given": ["equals", ["attr", "terms"], ["quote", "iPhone"]],
                "when": ["issue", ["query"] ],
                "then": [">",
                    ["size", ["filter", ["result"], 
                      ["lambda", ["and", 
                        ["matches", ["terms", [0]], ".*iPhone.*"], 
                        ["matches", ["terms", [0]], ".*シルバー.*"] ] ]
                    ]],
                    3
                ]
            }

Implement a feature to use 'regex' and 'fsm' of JCUnit8 as factors.

Implement a feature to use 'regex' and 'fsm' of JCUnit8 as factors.

Simple factor

  "factorSpace": {
    "factors": {
      "service": {
        "type":"simple",
        "args":[
          "shoppingmall",
          "bookstore"
        ]
      },

Regex factor

  "factorSpace": {
    "factors": {
      "bankAccount": {
        "type":"regex",
        "args":[
          "open(deposit|withdraw|getBalance)*"
        ]
      },

FSM factor

  "factorSpace": {
    "factors": {
      "flyingSpaghettiMonster": {
        "type":"fsm",
        "args":[
          "com.github.examples.fsm.FlyingSpaghettiMonsterSpec"
        ]
      },

Implement reporting featrure

Implement reporting feature where following report files will be generated

$ find reports -name report.json -exec sh -c 'echo filename {} && cat {} && echo' \;
filename reports/0/0/report.json
{"threshold":100,"valueOfF":50}
filename reports/0/1/report.json
{"threshold":100,"valueOfF":150}

if you write and execute this script.

{
  "description":"Simple",
  "factorSpace": {
    "factors":{"f":[50,150]}
  },
  "testOracles": [
    {
      "description": "Report example",
      "given": ["always"],
      "when": ["nop"],
      "then": [">",
        ["write_report", "valueOfF", ["attr", "f"]],
        ["write_report", "threshold", 100]
      ],
      "after": ["submit"]
    }
  ]
}