Do more per VM invocation about caliper

[GoogleCodeExporter]

note that this is a correctness issue -- caliper is currently reporting totally 
bogus 
results.

Original comment by [email protected] on 7 Jun 2010 at 5:48

• Changed title: Do more per VM invocation
• Added labels: Milestone-0.5

[GoogleCodeExporter]

Original comment by [email protected] on 14 Jan 2011 at 11:09

• Added labels: Milestone-1.0

[GoogleCodeExporter]

I can definitely tell that the order of warming up will affect HotSpot 
statistics and if different choices are made, the results will be different. 
This is the case we have in JUnitBenchmarks project -- the order of JUnit tests 
turned into benchmarks does affect the outcome (I only noticed this when 
compared the results against the same test executed in Caliper in separate VMs).

Original comment by [email protected] on 4 Mar 2011 at 4:43

[GoogleCodeExporter]

Original comment by [email protected] on 19 Mar 2011 at 2:13

• Added labels: Milestone-Post-1.0

[GoogleCodeExporter]

Original comment by [email protected] on 19 Mar 2011 at 3:06

• Added labels: Type-Enhancement

[GoogleCodeExporter]

Original comment by [email protected] on 8 Feb 2012 at 9:49

• Added labels: Component-Runner

[GoogleCodeExporter]

Original comment by [email protected] on 1 Nov 2012 at 8:32

[GoogleCodeExporter]

+martin is expressing some concern about this too.

Original comment by [email protected] on 11 Oct 2013 at 9:40

[GoogleCodeExporter]

I have also seen benchmark results highly dependent on the order of warmup - 
JIT optimizes for the profile collected during first method warmup.  That said, 
I would in general prefer to have my methods run in the same VM, to make 
execution less artificial.  So I support this change, but y'all had better vary 
warmup order.

Original comment by [email protected] on 11 Oct 2013 at 9:48

[GoogleCodeExporter]

I remain skeptical that such a change would make a benchmark less artificial.  
They would just be artificial in a a different way that is less predictable.  
We know that microbenchmarks are somewhat contrived anyway, so half-efforts 
toward making them "realistic" feel a little futile.

There's also a fair bit of work to be done to figure out how to make this work 
since the relatively simple estimation work that we do to guess at reps for 
microbenchmark warmup would need to be replaced.

That said, I see no reason to have it as an option if we can devise a strategy 
to make it work.

I should also mention that if our primary concern is total run execution time, 
the far easier target than this warmup business is just running instruments 
that aren't sensitive to resource constraints (e.g.: the allocation instrument) 
concurrently.

Original comment by [email protected] on 12 Oct 2013 at 2:37