Deploy machine learning with confidence.

NoRegressions is a tool for monitoring, evaluating, and testing machine learning models before production and after experimentation. The operating principle of NoRegressions is that all ML models are black boxes with certain inputs producing expected outputs. With NoRegressions you can apply previously unseen data as inputs to a model, and compare the results with expected outputs. NoRegressions is written in DotNet Core and runs on Windows, Mac and Linux.

NoRegressions provides a CLI that can:

• Manage Azure Blobs as data sources.
• Create labelled datasets from csv or directories.
• Test different model targets with your own datasets.
• Report results in ML FLow.

• Custom Vision Service
• ... more to come

On Windows:

iex ((New-Object System.Net.WebClient).DownloadString('https://raw.githubusercontent.com/JasonTheDeveloper/NoRegressions/master/install.ps1'))

On Linux:

wget -O - https://raw.githubusercontent.com/JasonTheDeveloper/NoRegressions/master/install.sh | bash

On Mac:

Homebrew is the easiest way to manage your CLI install. It provides convenient ways to install, update, and uninstall. If you don't have homebrew available on your system, install homebrew before continuing.

You can install the CLI by updating your brew repository information, and then running the install command:

brew update && brew tap AussieDevCrew/NoRegressions https://github.com/JasonTheDeveloper/NoRegressions.git && brew install AussieDevCrew/NoRegressions/noreg-cli

Now NoRegressions is aliased as noreg

On Mac:

brew remove AussieDevCrew/NoRegressions/noreg-cli

There are many more examples in the examples directory

• Help:

noreg help

• Upload a file to Azure Blob Storage:

noreg blob --upload --destination my_container --filepath "/path/to/my_image.jpg"

• Write URLs from my_container into list.txt

noreg blob --list-blob my_container --output list.txt

• Create a dataset called my_dataset

noreg create-dataset --id my_dataset

• Add images to a dataset. Currently we use a --type flag to help the test runner understand the test results.

noreg update-dataset --id my_dataset -l my_label --type SingleClassImage --from-file "list.txt"

• Run a test against a Custom Vision test target with my_dataset.

noreg test -t CustomVision -s my_dataset

You can configure the cli with noreg config

Tools are all DotNet Core, minimum version 2.2, which you can download here

• Build the CLI: dotnet build src/cli
• Run the tests: dotnet test test/unit

This project is very new and there is lots to do :-)

Currently, we are only supporting Custom Vision as a target for testing. We are also only supporting single label images.

• Add multi-class image recognition tests
• Add object-counting image recognition tests
• New targets (I don't know which)