Comments (10)
note that this is a correctness issue -- caliper is currently reporting totally
bogus
results.
Original comment by [email protected]
on 7 Jun 2010 at 5:48
- Changed title: Do more per VM invocation
- Added labels: Milestone-0.5
from caliper.
Original comment by [email protected]
on 14 Jan 2011 at 11:09
- Added labels: Milestone-1.0
from caliper.
I can definitely tell that the order of warming up will affect HotSpot
statistics and if different choices are made, the results will be different.
This is the case we have in JUnitBenchmarks project -- the order of JUnit tests
turned into benchmarks does affect the outcome (I only noticed this when
compared the results against the same test executed in Caliper in separate VMs).
Original comment by [email protected]
on 4 Mar 2011 at 4:43
from caliper.
Original comment by [email protected]
on 19 Mar 2011 at 2:13
- Added labels: Milestone-Post-1.0
from caliper.
Original comment by [email protected]
on 19 Mar 2011 at 3:06
- Added labels: Type-Enhancement
from caliper.
Original comment by [email protected]
on 8 Feb 2012 at 9:49
- Added labels: Component-Runner
from caliper.
Original comment by [email protected]
on 1 Nov 2012 at 8:32
from caliper.
+martin is expressing some concern about this too.
Original comment by [email protected]
on 11 Oct 2013 at 9:40
from caliper.
I have also seen benchmark results highly dependent on the order of warmup -
JIT optimizes for the profile collected during first method warmup. That said,
I would in general prefer to have my methods run in the same VM, to make
execution less artificial. So I support this change, but y'all had better vary
warmup order.
Original comment by [email protected]
on 11 Oct 2013 at 9:48
from caliper.
I remain skeptical that such a change would make a benchmark less artificial.
They would just be artificial in a a different way that is less predictable.
We know that microbenchmarks are somewhat contrived anyway, so half-efforts
toward making them "realistic" feel a little futile.
There's also a fair bit of work to be done to figure out how to make this work
since the relatively simple estimation work that we do to guess at reps for
microbenchmark warmup would need to be replaced.
That said, I see no reason to have it as an option if we can devise a strategy
to make it work.
I should also mention that if our primary concern is total run execution time,
the far easier target than this warmup business is just running instruments
that aren't sensitive to resource constraints (e.g.: the allocation instrument)
concurrently.
Original comment by [email protected]
on 12 Oct 2013 at 2:37
from caliper.
Related Issues (20)
- Build is falling : Dagger and dagger-compiler jar is not being downloaded HOT 3
- frequent releases to maven repo (at least snapshot) HOT 8
- failing tests (f8dcb9a794b0cee6f9408d1b692572dbcd03788c) HOT 3
- scripts/caliper broken
- StreamServiceTest is flaky
- A simple example of how to use HOT 2
- Caliper with HttpServletRequest
- Missing pom.xml from recent commit
- Results should not be uploaded to the web by default HOT 1
- @Benchmark annotation
- -XX:CICompilerCount=2 vmOption does not apply
- microbenchmarks link in the home page of caliper wiki is not right HOT 1
- Fix Caliper tests under Java 10+11 HOT 3
- Unreleased Resource: Streams HOT 1
- Check for problems with ByteBuffer covariant return types when building with JDK 9+ HOT 2
- Maven release very old (2015)
- There is a vulnerability in OkHttp 2.5.0,upgrade recommended
- There is a vulnerability in Guava: Google Core Libraries for Java 28.1-jre,upgrade recommended
- Say something about (a) JMH and (b) the Jetpack Benchmark library HOT 7
- [Docs] Possible outdated documentation
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from caliper.