leifwalsh / perfume

Interactive performance benchmarking in Jupyter

perfume

Interactive performance benchmarking in Jupyter

• Free software: BSD license
• Documentation: https://perfume.readthedocs.io.

Overview

perfume is a performance benchmarking tool that provides quick feedback on the systems under test.

The primary goals are:

• Prioritize analysis of distributions of latency, not averages.
• Support both immediate feedback and robust benchmarking with many samples, through a UI that updates as we collect more information.
• Provide raw data back to the user, for flexible custom analysis.
• Provide helpful post-processing analysis and charting tools.

Features

• Live-updating histogram chart and descriptive statistics during a benchmark run.
• Jupyter notebook integration.
• Tunable benchmarking overhead.
• Comparative analysis of multiple functions under test.
• Powerful post-processing analysis tools.

Demo

You can check out an example notebook using perfume.

Installing

pip install perfume-bench

Credits

This package was created with Cookiecutter and the audreyr/cookiecutter-pypackage project template.

perfume's Issues

Add an observation time to each sample.

this would allow users to see distribution as-of some time in the experiment, plot statistics over time, etc.

to Leif's point, the observations are ordered so this is possible but I like the idea of having this in time semantics. for example, i could show that mean runtime over time, t, changed by X.

dataframe outputs for descriptive stats and k-s tests update old output cells

Consider alternatives to linear equal-width histogram bins

Is your feature request related to a problem? Please describe.

Linear equal-width histogram binning has some known problems for interpretability. They can oversmooth in regions of high density (and we may expect latency measurements to have such regions), and have trouble accurately representing outliers, as one outlier compresses the rest of the chart dramatically.

Describe the solution you'd like

I would like to explore alternatives to this histogram approach. I'm not sure what the right solution is yet.

Describe alternatives you've considered

Some alternatives:

• Equal-area histogram bins
• Log-scale equal-width histogram bins
• Equal-width histogram bins on log-transformed values
• Scrap histograms altogether, just use the kernel density estimation
• Scrap histograms altogether, just plot empirical CDFs

resolve conflict with the other perfume

I created a new conda env and then did 'pip install perfume' , instead of 'pip install ./perfume' where I had cloned this perfume locally.

Conda happilly installed the other perfume
https://github.com/hoh/perfume

not sure if this is a problem . but be aware.

suggestion: analyze whether two results are different

it'd be cool to have Perfume do a k-s test or something similar and report that two benchmarks are/are not significantly different from each other.

Add setup functionality

For some benchmarks, we may want to run (but not time) some initialization code before each run