[question] How to change the number of samples? about criterion.rs

[bheisler]

You can change that value using the sample_size method on Criterion:

let mut criterion = Default::default();
criterion.sample_size(150);
criterion.bench_function(...)

Please forgive any typos, I'm writing this from my phone so I can't easily check this code.

Having said that, could you explain why you want to change that option?

[dvdplm]

Having said that, could you explain why you want to change that option?

In my case I'm measuring writes to Redis and I'm interested in seeing how my code behaves as memory fills up, the hypothesis being that the system will get slower as the data set size increase.

Maybe I'm misunderstanding something though: what is, exactly, a "sample"? It is not the number of iterations I take it?

[bheisler]

Well, the most direct way to increase the number of iterations is to increase the measurement time (see here).

I'd be interested to hear how well Criterion.rs works for you in this case. For best results, the time per iteration should be roughly independent of the number of iterations. A better approach might be to create a second benchmark where you pre-populate a large database for each sample and run insertions on that. In that case, the difference between small and large iteration counts will be smaller than it would if you started with an empty database. On the other hand, there are a lot of things one might want to benchmark that won't be independent of the iteration count, so it could maybe support that use case better somehow. I'll have to think on that some more.

That's correct, a sample is not the same as an iteration. The number of iterations isn't directly controllable; Criterion.rs attempts to estimate how many iterations will result in the specified measurement time (where possible), then divides those iterations up into N samples. I'm working on some documentation to explain this all in more detail.

[dvdplm]

then divides those iterations up into N samples. I'm working on some documentation to explain this all in more detail.

That makes sense, thank you for the explanation and +1 at more docs, I think this library has a great future, loving it. :)