jdrew1303 / confidence

Confidence.js: Make sense of your A/B test results

Confidence.js

Confidence.js is a light-weight JavaScript library to help you make sense of your A/B Test results. Given an A/B test (or "Split Test") result set, Confidence.js will tell you if a statistical "winner" can be determined.

Documentation

• API Reference and Examples
• How It Works

Getting Started

Download confidence.js here

Include confidence.js in your HTML.

<script src="path/to/confidence.js"></script>

You're all ready! Start testing... with confidence.

Example

We have many examples in the API Reference. Here's a simple one to get you started:

// Create a new test result
var myConfidence = new Confidence();

// Add variant A results
myConfidence.addVariant({
  id: 'A',
  name: 'Variant A',
  conversionCount: 2500,
  eventCount: 3000
});

// Add variant B results
myConfidence.addVariant({
  id: 'B',
  name: 'Variant B',
  conversionCount: 1500,
  eventCount: 3000
});

// Get results, using Z-Test method
zTestResult = myConfidence.getResult();
/*
{
  hasWinner: true,
  hasEnoughData: true,
  winnerID: 'A',
  winnerName: 'Variant A',
  confidencePercent: 95.00,
  confidenceInterval: { min: 82, max: 84.67 },
  readable: 'With 95% confidence, the true population
    parameter of the "Variant A" variant will
    fall between 82% and 84.67%.'
}
*/

How It Works

Confidence.js provides two methods for evaluating an A/B test:

• The Z-Test Method
• Chi Square Test and Marascuillo's Procedure

Selecting an appropriate method depends on your data set and use case - both have advantages.

Learn how each method works here.

Running the Tests

npm install
npm test

Feature Requests and TODO

• implement "Confidence to beat baseline" comparison (in progress)
• add removeVariant function

Issues and Questions

Found a bug? Create an issue here on GitHub!

For general questions, tweet me @jessicaraygun.

Authors

Developed and maintained by Jessica Thomas, Data Scientist @ sendwithus.com, with guidance and tutelage from Statistics Mastermind Emily Malcolm.