Use the same benchmark suite when judging about pkgbenchmark.jl

[shashi]

Yeah, this is a problem. But it might be the case that old benchmarks might not run on the most recent commit... For such situations, you'd have to create a new benchmark file and pass that as script.

[MilesCranmer]

Sorry if this is already solved, I couldn't find this anywhere.

Could this be done automatically inside benchmarkpkg?
e.g., the simplest solution, assuming there's no includes would be:

tmp_script = ".pkgbenchmark" * script
cp(script, tmp_script; force=true)

at the start of evaluation, and then run on tmp_script when comparing to older commits. My guess is it is quite common to compare to older commits, written before the benchmark was created or modified. So it would be nice to have some flag that could set this up automatically.

Thoughts?

Cheers,
Miles