Add a way to programmatically get benchmark results about criterion.rs

[japaric]

#15 is about letting the user run a custom analysis pass and for that we need to exposes the analysis methods (outliers, bootstrap, etc) as functions.

I think this issue should be prioritized over #15.

[dignifiedquire]

This would be great if this could be done, happy to help. I am currently resorting to parsing the stdout which is not ideal.

My use case is, that I want the analyzed data shipped to a database so I can run easy comparisons over time.

[fredpointzero]

I would really appreviate this feature, I am currently using Criterion for automated performance regression tested, but parsing the output is not a reliable way to do it.

[fzyzcjy]

Hi this is very appealing feature! I wonder when can we use this?

[NoraCodes]

I just wanted to hop in to ask for this as well. Currently, I'm using Criterion to compare several algorithms for a particular workload, and being able to more easily collect, correlate, and plot the output of multiple charts would be extremely useful.

As it is, I have to either parse the cargo bench output, scrape the HTML reports, or use the unstable machine-readable output - none of which are attractive options. Ideally I'd be able to grab the output of the benchmarks by, say, passing a custom function to criterion_main!().

Is this something the project is interested in supporting? I'd be happy to write up a pull request if so.

[lemmih]

It would be most welcome. Significant architectural changes are probably needed, though.

[theJasonFan]

Not sure how helpful this would be, but one workaround would be to do something like:
https://gist.github.com/theJasonFan/65fa3e514a7fe7b179412f41f7c7168b

And invoke bench_function(...) in a binary.

It's quite hacky though. Note that criterion expects to know all group and benchmark names before any benchmark is run to generate plots that compare performance. So running benchmarks this way and generating benchmark names programmatically via command line input might break things.

I think one could direct the output to a different directory with:

criterion.rs/src/lib.rs

Line 700 in b61121b

pub fn output_directory(mut self, path: &Path) -> Criterion<M> {

.