classicvalues / ep-stats

Statistics for Experimentation Platform

ep-stats

Statistical package for the experimentation platform.

It provides a general Python package and REST API that can be used to evaluate any metric in an AB test experiment.

Features

• Robust two-tailed t-test implementation with multiple p-value corrections and delta methods applied.
• Sequential evaluations allow experiments to be stopped early.
• Connect it to any data source to get either pre-aggregated or per randomization unit data.
• Simple expression language to define arbitrary metrics.
• REST API to integrate it as a service in experimentation portal with score cards.

Documentation

We have got a lovely documentation.

Base Example

ep-stats allows for a quick experiment evaluation. We are using sample testing data to evaluate metric Click-through Rate in experiment test-conversion.

from epstats.toolkit import Experiment, Metric, SrmCheck
experiment = Experiment(
    'test-conversion',
    'a',
    [Metric(
        1,
        'Click-through Rate',
        'count(test_unit_type.unit.click)',
        'count(test_unit_type.global.exposure)'),
    ],
    [SrmCheck(1, 'SRM', 'count(test_unit_type.global.exposure)')],
    unit_type='test_unit_type')

# This gets testing data, use other Dao or get aggregated goals in some other way.
from epstats.toolkit.testing import TestData
goals = TestData.load_goals_agg(experiment.id)

# evaluate experiment
ev = experiment.evaluate_agg(goals)

ev contains evaluations of exposures, metrics, and checks. This will provide the following output.

ev.exposures:

exp_id	exp_variant_id	exposures
test-conversion	a	21
test-conversion	b	26

ev.metrics:

exp_id	metric_id	metric_name	exp_variant_id	count	mean	std	sum_value	confidence_level	diff	test_stat	p_value	confidence_interval	standard_error	degrees_of_freedom
test-conversion	1	Click-through Rate	a	21	0.238095	0.436436	5	0.95	0	0	1	1.14329	0.565685	40
test-conversion	1	Click-through Rate	b	26	0.269231	0.452344	7	0.95	0.130769	0.223152	0.82446	1.18137	0.586008	43.5401

ev.checks:

exp_id	check_id	check_name	variable_id	value
test-conversion	1	SRM	p_value	0.465803
test-conversion	1	SRM	test_stat	0.531915
test-conversion	1	SRM	confidence_level	0.999000

Installation

You can install this package via pip.

pip install ep-stats

Running

You can run a testing version of ep-stats via

python -m epstats

Then, see Swagger on http://localhost:8080/docs for API documentation.

Contributing

To get started locally, you can clone the repo and quickly get started using the Makefile.

git clone https://github.com/avast/ep-stats.git
cd ep-stats
make install-dev

It sets a new virtual environment venv in ./venv using venv, installs all development dependencies, and sets pre-commit git hooks to keep the code neatly formatted with flake8 and brunette.

To run tests, you can use Makefile as well.

source venv/bin/activate  # activate python environment
make check

To run a development version of ep-stats do

source venv/bin/activate
cd src
python -m epstats

Documentation

To update documentation run

mkdocs gh-deploy

It updates documentation in GitHub pages stored in branch gh-pages.

Inspiration

Software engineering practices of this package have been heavily inspired by marvelous calmcode.io site managed by Vincent D. Warmerdam.