gior / proficiency

Regression test tool for rasa_nlu

Proficiency

Regression test tool for rasa_nlu.

What is it for?

Proficiency allows bot developers to stay confident of the bot behaviour while its language model grows and becomes more complex. Developers can

• define what intent and entites are expected given a certain query
• define confidence levels required for certain, acceptable and uncertain matches
• get a visual report of the test run
• see details of the issues

Warning

In order to understand the following instructions you should be familiar with Rasa NLU and its terms. Please take a look at the docs if you feel you're missing something.

Setup

Install the library

npm install -g proficiency

Create a project folder where you will store

• config.json: the configuration file, and
• suite.json: the test suite.

For your convenience the folder src/sample-project contains example files that you can copy and customize.

Sample configuration is

{
  "project": "new_bot",
  "endpoints" : {
    "dev": {
      "name": "Dev",
      "url": "http://localhost:5000/parse"
    },
    "prod": {
      "name": "Production",
      "url": "https://example.com:5000/parse",
      "token": "rasa_nlu_server_token_if_any"
    }
  },
  "highConfidence": 0.75,
  "mediumConfidence": 0.65
}

As you can see, you can specify connection tokens for your endpoints. If null or missing, no token will be sent. Note: Be sure to use the same project name you used in Rasa. Otherwise the NLU serivce will be unable to answer.

Write regression tests in suite.json.

{
  "examples": [
    {
      "sentence": "Hi!",
      "expected": {
        "intent": {
          "name": "greetings"
        },
        "entities": []
      }
    },  
    {
      "sentence": "Hi Sarah!",
      "expected": {
        "intent": {
          "name": "greetings"
        },
        "entities": [
          {
            "start": 3,
            "end": 8,
            "value": "Sarah",
            "entity": "name"
          }
        ]
      }
    }
  ]
}

The format of each example is compatible with rasa_nlu responses. This way you can add new examples by

• filling the sentence attribute, then
• copying and pasting intent and entities from the response you received from rasa_nlu service, then
• possibly removing fields like confidence, extractor, processors. They are not useful in this file.

Look at the available options

proficiency -h

Launch the test suite

Using your development endpoint:

npx proficiency <regression test folder path>

In production:

npx proficiency <regression test folder path> -p

Happy testing!