Comments (7)
I think we should do this after the transition from Makefile to cmake, as all the components will be easier to integrate.
I think that it would be better to implement this in the following order:
- write the micro benchmarks with the google benchmark framework;
- produce the graphic visualization.
I am happy for you to use Java+gnuplot, but I think that it would be better to use python+mathplotlib, as it might be easier to maintain and run. It will require less configuration than setting up a Java VM, and I think it will be easier to deploy on Travis.
from sleef.
I am now considering to overhaul the benchmarking tools.
My plan is to automate graphing with gnuplot first, and then introduce Google's framework.
In order to process data, I am planning to use java, since I am used to it.
from sleef.
Do we need to deploy it on Travis?
You know, the benchmark results cannot be trusted since Travis is on a cloud.
The problem with python is that I need time to learn those things.
I have lots of experience with Java, and some experience with gnuplot, so it would be much easier for me.
from sleef.
Then, Java + gnuplot is, you are right, we cannot run benchmarks on the cloud.
I still think you should first use the micro benchmark system of google and then plot the data, because the output of Google benchmark might be different from what you expect in the plotting scripts. You might need to rework the scripts if that is going to be the case. Also, Google benchmark has some report facilities (see [1] for example), which might be used instead of the plotting tool.
I recommend you to leave the scripts for plotting as the last (optional) part of this work.
[1] http://1.bp.blogspot.com/-wk7hsdYodo8/UtS75FZag6I/AAAAAAAAAks/cQFiCXPbtwk/s1600/image00.png
from sleef.
Could you explain a little bit about how you are planning to use the output data?
I have been always thought this work as a part of writing our paper.
From that perspective, drawing graph is essential.
from sleef.
Getting numbers is very important. Using google framework should guarantee that we have reliable evaluations. If the aim of setting up this is only to get the numbers in the paper, I think we don't need to store any script in here for that.
My goal here was to make sure that the numbers we were getting where reliable, and I believe that those we could get with google micro benchmarks framework have such property. They would be reliable not only for us, but also for people reading these numbers are they are produce via a standard tool.
Also, using google micro benchmarks would make easier for other people to verify our claims on their own machines.
For the paper, maybe we could store the scripts that generate the graphics in the paper itself?
from sleef.
I still don't understand why there could be so much difference in reliability of measured values. We are just measuring execution time of small C codes, which basically don't have conditional branch or memory access. Execution time is highly reproducible. If this were java, there would be many things to consider like JIT or garbage collection.
How much reliability do you need?
from sleef.
Related Issues (20)
- Test more OS-es in Github Actions HOT 7
- Add GNU make as potential generator in CI
- Please, create tag release HOT 3
- dft test gets stuck during initialisation when hardware vector length is very long HOT 2
- Failure on ppc64 when cross compiled with gcc-11 toolchain
- Inaccuracy for f32 erf on non-FMA instruction set
- New test failures in 3.6 HOT 7
- F64 exp returns infinity slightly too soon
- RISC-V architecture missing on https://sleef.org/ HOT 1
- Clang on Windows and GNU ABI
- RISC-V: exploit `-mrvv-vector-bits=zvl` when used HOT 1
- Missing unversioned symbolic link for GNUABI version in 3.6 HOT 3
- A few API functions were (accidentally?) removed in 3.6 HOT 11
- Integrate ARM-software/optimized-routines into sleef? HOT 2
- Improve portability of libm and quad testers
- Weird linker errors that don't make sense HOT 7
- No rule to make target '/bin/mkdisp' crosscompile using sleef/toolchains/aarch64-gcc.cmake HOT 2
- can not compile under Visual Studio 17 2022 / ARM64 HOT 5
- Building sleef quad inline for CUDA problem HOT 4
- Feature detection is not thread-safe HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from sleef.