libnonius / nonius Goto Github PK

View Code? Open in Web Editor NEW

357.0 357.0 47.0 1.55 MB

A C++ micro-benchmarking framework

Home Page: https://nonius.io

License: Creative Commons Zero v1.0 Universal

C++ 56.38% Smarty 42.45% CSS 0.15% JavaScript 0.34% Python 0.68%

benchmarking c-plus-plus

nonius's People

Contributors

Stargazers

Watchers

nonius's Issues

nonius.io does not load in Safari on OSX 10.12

When I go to https://nonius.io in safari on OSX 10.12 beta 3 the website fails to load with the error shown below:

Make nonius available in Nuget

Make it available for VS using native Nuget package

nonius.io leaks referrers

See https://webbkoll.dataskydd.net/en/results?url=https%3A%2F%2Fnonius.io%2F

Adjust macro interface for generating benchmarks

The interface for automatically register benchmarks is the following:

NONIUS_BENCHMARK("My Benchmark", [](nonius::chronometer meter)
{
//...
})

Because of that it is not possible to use #ifdef inside the benchmark itself.
it looks also not nice, see the end of the braces? }) This makes it harder for the client writing benchmarks.

I came up with following idea:

NONIUS_BENCHMARK("Group Sorting", "Sort 1")
{
    meter.measure([&]() 
    {
       // benchmark code
    }
}

You can look in the testing framework catch, which you already use, how it is implemented there, but it's pretty easy. Basically the macro has to expand to a temporary function which gets as argument this nonius::chronometer meter

Additionally:
Like you can see, I would like to group benchmarks together, but this is another big request, but you should keep this in mind when redesigning the interface.

Benchmark times depend on size of vector outside of measurement

This is a typical benchmark code I use with nonius:

nonius::benchmark my_benchmark()
{
    return nonius::benchmark("title", [](nonius::chronometer meter)
    {
       // here I do some heavy setup; before actually running the benchmark
        std::vector<int> foo(10);
        meter.measure([&]()
        {
            my_method()
        });
    });
}

I've increased the size of the std::vector to 1000, and this has lead to higher sample values, although the object is not used at all. For my point of view I always thought nonius will measure only the time of the function call meter.measure([&](). But this seems to be not the case.
Is this a bug or a feature? Or am I totally wrong?

Autocorrelation

Autocorrelation analysis could help identify problems with benchmarks, or simply interesting behaviour in some code.

Boost Overflow Error

Running benchmarks on my machine usually^† ends up in an internal error:

benchmarking to_string(42)
collecting 100 samples, 3 iterations each, in estimated 1173 μs
PANIC: clock is on fire
Error in function boost::math::erfc_inv<double>(double, double): Overflow Error

Upon looking it was found out that the issue had to do with prob_n being zero which led to an overflow error.

The reason for prob_n being zero is because resample was an array full of negative values (-10) and none were different.

Upon further inspection, it turned out that the samples vector was also full of negative values for some reason. Which isn't supposed to happen. Looking into what made the samples negative led to this code:

auto elapsed = model.finished - model.started;
return ((elapsed - env.clock_cost.mean) / plan.iterations_per_sample);

It just so happens that elapsed ends up being 0 too or somehow less than env.clock_cost.mean which makes the value negative setting the whole explosion above to happen.

Using Boost.Chrono instead of std::chrono seems to fix the issue. The true resolutions for each:

Boost.Chrono: clock resolution: mean is 38.322 ns (10240002 iterations)
std::chrono: clock resolution: mean is 11.6849 ns (40960002 iterations)

However I don't really think std::chrono is the issue since it's able to have nanosecond resolution required.

N.B.: You've given yourself a note:

I do need to guard against that, though. Since an empty run (i.e., just the cost of the clock) can possibly run faster than the mean clock cost, I need to clamp to 0.
Clamp to zero, and guard against uniform samples.

_{†: Sometimes it doesn't give this error but the statistics seems wrong. Using --no-analysis seems to work without triggering the error.}

Support benchmark parameters

If benchmarks can be parameterized, once can run the same code over inputs of different sizes, for example, and check how some algorithm scales with input.

Markdown table generator

It will be nice to have output generator for markdown tables. (esp with sorting options).

faster_than_clock.c++ test fail on Windows

test/faster_than_clock.c++:22: FAILED:
due to unexpected exception with message:
  could not measure benchmark, maybe it was optimized away

This test is failing for me when compiled with msvc2015 and mingw64 g++. I've tried it on multiple machines. It seems to be OS specific rather than compiler or hardware.

"Submodule the boost stuff"

Per this

Goodness of fit for varying parameters

Given #36, it is only natural to automate verification of fitness to different models, in order to estimate the likelihood of some code belonging to a particular complexity class.

Shuffle benchmarks

Hot code performs differently from cold code. Caches and stuff.

Currently execution order goes like this: benchmark0/sample0, benchmark0/sample1, ..., benchmark0/sampleN, benchmark1/sample0, benchmark1/sample1, ..., benchmark1/sampleN, ..., benchmarkM/sample0, benchmarkM/sample1, ..., benchmarkM/sampleN. This means that all executions of a single benchmark are sequenced together, which can lead to differences from caching effects.

It should be possible to execute different orders, like: benchmark0/sample0, benchmark1/sample0, ..., benchmarkM/sample0, benchmark0/sample1, benchmark1/sample1, ..., benchmarkN/sample1, ..., benchmark0/sampleN, benchmark1/sampleN, ..., benchmarkM/sampleN. This would alternate sampling between different benchmarks and reduce cache effects of locality.

Random ordering is also a possibility.

Add Continuous Integration support for nonius

The current release (1.1.0) will bring up again compiler warnings for VSx64.

// line 1541, the 'n' variable is std::size_t
double outlier_variance = detail::outlier_variance(mean_estimate, stddev_estimate, n);

To avoid these kind of problems in the feature, please add support for CI for nonius:
See following template as starting point: https://github.com/headmyshoulder/cit
For Visual Studio use: http://www.appveyor.com/

Build failed with gcc4.9.3

Compiling a tiny test project consisting only of example1.c++, I get the following error:

In file included from /home/claravu/nonius/include/nonius/configuration.h++:17:0,
from /home/claravu/nonius/include/nonius/benchmark.h++:18,
from /home/claravu/nonius/include/nonius/nonius.h++:18,
from /home/claravu/nonius/include/nonius/nonius_single.h++:14,
from /home/claravu/test_nonius/example1.c++:36:
/home/claravu/nonius/include/nonius/param.h++: In member function ‘nonius::param nonius::param::parse(const string&) const’:
/home/claravu/nonius/include/nonius/param.h++:103:38: error: use of deleted function ‘std::basic_stringstream::basic_stringstream(const std::basic_stringstream&)’
auto ss = std::stringstream{s};

Changing the offending line to:

  std::stringstream ss;
  ss.str(s);

Made the error go away, but I'm not 100% sure this it true to the intent. The error does not occur in v1.1.2.

I'm trying out nonius for benchmarking on a new project, looks pretty good so far!

Support benchmark suites

Grouping benchmarks into suites enables certain functionality like having different sets of parameters for different benchmarks.

Bug in samples

It looks like that in 3rd sample and up code:

#define NONIUS_RUNNER
#include <nonius.h++>

must be replaced with:

#define NONIUS_RUNNER
#include <nonius/nonius_single.h++>

Document Plotly

Now that #54 is merged, the documentation should reflect that highcharts is no longer used.

Add build instructions

Some instructions about using the build scripts for people wanting to tweak or contribute to nonius are needed.

Document parameters

The documentation needs to be updated for #56 and #63.

Officially support MSVC 19.0 (VS2015)

MSVC should be added to the build system for automatic testing.

Freezing on fast benchmarks

I have encountered a problem after updating nonious to the latest version of branch devel. Benchmarks that supposed to run very fast cause a freezing. For example

#include <memory>

#include <nonius/nonius.h++>
#include <nonius/main.h++>

NONIUS_BENCHMARK("shared_ptr<int> default constructor", [](nonius::chronometer meter)
{
    nonius::storage_for<std::shared_ptr<int>> storage;
    meter.measure([&storage] {
        storage.construct();
    });
});

produce the following output and then freezes:

clock resolution: mean is 16.1415 ns (40960002 iterations)
benchmarking shared_ptr default constructor

I guess that the bug appeared after latest commits in may/june because same benchmark had been running successfully before.

I compile it with gcc 4.9.3 on ubuntu 14.04. My machine is x64 Intel® Core™ i7-4790 CPU @ 3.60GHz × 8 with 16 Gb RAM.

Document new benchmark syntax

The changes in #65 need to be documented.

Benchmark failed, because it runs too fast

I have a problem on a PC, which is quite fast. A benchmark failed, because it runs too fast.

The benchmark basically measure the time of a simple function call (a void function which does nothing)

obj.method_call();

On another machine I can run the benchmark (mean: 0.3ns).

How should I handle this kind of problem in general?

Another problem which I've seen is: For certain benchmarks I see sometimes sample ranges alternating between 0.0 ns and 300 ns. Is this a known problem? What can I do about it?

To get a correct result, I have to run the benchmark twice.

nonius.io makes third-party requests

See https://webbkoll.dataskydd.net/en/results?url=https%3A%2F%2Fnonius.io%2F

Document nonius::main

Now that #26 was merged, nonius::main should be part of the documented interface.

Collect additional metrics

Sometimes things other than time need to be measured. Nonius should support collecting additional metrics like, e.g., cache misses and branch mispredictions. Ideally this would be a generically extensible feature that users can use to collect any custom metrics they desire.

Kernel density estimation

Support KDE outputs.

Websites not responding

The websites http://nonius.io/ and http://flamingdangerzone.com/nonius are currently not loading for me. Everything else works fine.

Number of iterations is not determined using chronometer::measure() duration

In the benchmarks below, to_string(4.2) with sleep always runs with 1 iteration, while to_string(4.2) with no sleep runs with around 100 iterations. This is because nonius::benchmark::prepare() computes the number of iterations based on the duration of the entire benchmark function, not the duration of the measure() function. This is a problem if the code prior to the measure() is expensive. In this example, the reported mean of the sleep version is 5.5 _micro_seconds, while the mean of the no-sleep version is 196 _nano_seconds!

#define NONIUS_RUNNER
#include <nonius.h++>
#include <thread>
#include <chrono>

NONIUS_BENCHMARK("to_string(4.2) with sleep", [] (nonius::chronometer chronometer) {
  std::this_thread::sleep_for(std::chrono::milliseconds(1));

  return chronometer.measure([] {
    return std::to_string(4.2);
  });
})

NONIUS_BENCHMARK("to_string(4.2) with no sleep", [] (nonius::chronometer chronometer) {
  return chronometer.measure([] {
    return std::to_string(4.2);
  });
})

Collect system information

Nonius should make a best effort attempt to collect useful available system information.

How to ignore destruction of object?

I want to ignore in the measurement the destruction of an object. I know the special objects: nonius::storage_for & nonius::destructable_object

But I think it will not help me, I have following case:

MyObject obj = method_generate_myobject()

The method method_generate_myobject() will generate an instance of MyObject with automatica storage. The method has access to a private ctor which I cannot invoke directly.
How can I ignore with nonius the destruction of MyObject during the benchmark run?

The build script should more easily support different compilers/stdlibs/boost versions

timeout_error encountered for (?) fast tests

I wanted to compare to ways of checking whether a boost variant is empty

http://stackoverflow.com/questions/31320857/how-to-determine-if-a-boostvariant-variable-is-empty/31322574#31322574

However, the which()==0 variant consistently eludes the benchmark algorithm by taking too much time to result in useful measurements. The name of testcase is which and the benchmark results in the following messages:

benchmarking which
which failed to run successfully
benchmark aborted

As you suggested in the lounge:

(And I like that nonius has trouble benchmarking such code because it tells you that even if you coax some results out of it, they might be meaningless)

it might indicate code that's hard to benchmark.

Could you recommend a way to get useful comparison here? Perhaps a rearrangement or a tweak (warmup_time e.g.).

I'd actually be happy if I could clearly see that the test fails to "converge" because it always takes negligible time - because then we have a clear winner if the other variant runs in non-negligible time. I guess in this respect this issue is close to Issue #14

Here's my test program

#include <boost/variant.hpp>
#include <nonius/benchmark.h++>
#include <nonius/main.h++>

namespace detail {
    struct is_blank : boost::static_visitor<bool> {
        constexpr is_blank() = default;

        constexpr bool operator()(boost::blank const&) const { return true; }

        template<typename T>
            constexpr bool operator()(T const&) const { return false; }
    };
}

template <typename... Ts>
constexpr bool is_blank(boost::variant<Ts...> const& v) {
return boost::apply_visitor(detail::is_blank(), v);
}

#include <algorithm>

namespace /*statics*/ {
    // initialization phase not part of benchmarks
    using V = boost::variant<boost::blank, bool, int>;
    static auto const test_data = [] {
        std::vector<V> many;
        std::generate_n(back_inserter(many), 1ul<<12, []()-> V { 
            V r;
            switch (rand()%3) {
                case 1: r = 42;    break;
                case 2: r = false; break;
            }
            return r; // NRVO (not critical)
        });
        return many;
    }();
}

NONIUS_BENCHMARK("apply_visitor", [](nonius::chronometer c) {
    c.measure([&](int i) {
            return is_blank(test_data[ i % test_data.size() ]);
        });
})

NONIUS_BENCHMARK("which", [](nonius::chronometer c) {
    //for (auto & el : test_data) std::cout << (0 == el.which()) << " ";
    //std::cout << '\n';
    c.measure([&](int i) {
            return (test_data[ i % test_data.size() ].which() == 0);
        });
})

Typical output:

$ ./test
clock resolution: mean is 17.7222 ns (81920002 iterations)

benchmarking apply_visitor
collecting 100 samples, 1148 iterations each, in estimated 1722 μs
mean: 13.6744 ns, lb 13.541 ns, ub 14.0019 ns, ci 0.95
std dev: 1.05495 ns, lb 0.508851 ns, ub 1.72528 ns, ci 0.95
found 9 outliers among 100 samples (9%)
variance is severely inflated by outliers

benchmarking which
which failed to run successfully
benchmark aborted

Officially support clang 3.7

Clang 3.7 is out. The build system should automatically test it.

Reinvent boost statistical functions

https://github.com/rmartinho/nonius/blob/devel/include/nonius/detail/stats.h++ uses boost::math::quantile, boost::math::normal, and boost::math::cdf.

These need to be replaced with non-boost equivalents to remove the boost dependency.

Reinvent boost string algorithms

https://github.com/rmartinho/nonius/blob/devel/include/nonius/detail/cpptempl.h uses boost::lexical_cast, boost::trim_copy_if, boost::trim_copy, boost::is_space, boost::split, and boost::starts_with. If the boost dependencies are to go away, these need to be replaced with non-boost equivalents.

Conan package for Nonius

I've created a Conan-package for Nonius here. It works for me on Mac OS X, but I have no way of testing it on Windows.

Also, in order to configure the Boost dependency better, it would be useful to know for which versions of Visual Studio specifically it is required.

I'm happy for any feedback. If you have no objections, once it's confirmed working on Windows, I'd publish it in the Conan central repository.

Explain environmental artifacts

The benchmark authoring guide needs to explain that the user is still responsible for certain characteristics of the environment that nonius cannot control nor account for, and also explain the extent of nonius' interaction with those.

undefined reference to `main'

Hey,

I've tried to compile the examples but I'm getting undefined reference to main
git log shows : 78df23a

g++ -o example2 example2.c++ --std=c++14 -I/home/rsoeldner/custom/nonius/include -I/home/rsoeldner/custom/nonius/include/nonius -lpthread
/usr/lib/../lib64/crt1.o: In function `_start':
/home/abuild/rpmbuild/BUILD/glibc-2.19/csu/../sysdeps/x86_64/start.S:118: undefined reference to `main'
collect2: error: ld returned 1 exit status

Added dependency on Boost Chrono?

Moving from rev 74b2de7 to ce74266

breaks linking my nonius test program with link errors:

/tmp/direct-simple-8546e9.o: In function `nonius::timing<boost::chrono::steady_clock::duration, nonius::detail::complete_type<std::result_of<nonius::detail::repeater<nonius::now<boost::chrono::steady_clock> > (int)>::type>::type> nonius::detail::run_for_at_least<boost::chrono::steady_clock, nonius::detail::repeater<nonius::now<boost::chrono::steady_clock> > >(boost::chrono::steady_clock::duration, int, nonius::detail::repeater<nonius::now<boost::chrono::steady_clock> >&&)':
direct-simple.cpp:(.text._ZN6nonius6detail16run_for_at_leastIN5boost6chrono12steady_clockENS0_8repeaterINS_3nowIS4_EEEEEENS_6timingINT_8durationENS0_13complete_typeINSt9result_ofIFT0_iEE4typeEE4typeEEESB_iOSE_[_ZN6nonius6detail16run_for_at_leastIN5boost6chrono12steady_clockENS0_8repeaterINS_3nowIS4_EEEEEENS_6timingINT_8durationENS0_13complete_typeINSt9result_ofIFT0_iEE4typeEE4typeEEESB_iOSE_]+0x15): undefined reference to `boost::chrono::steady_clock::now()'
direct-simple.cpp:(.text._ZN6nonius6detail16run_for_at_leastIN5boost6chrono12steady_clockENS0_8repeaterINS_3nowIS4_EEEEEENS_6timingINT_8durationENS0_13complete_typeINSt9result_ofIFT0_iEE4typeEE4typeEEESB_iOSE_[_ZN6nonius6detail16run_for_at_leastIN5boost6chrono12steady_clockENS0_8repeaterINS_3nowIS4_EEEEEENS_6timingINT_8durationENS0_13complete_typeINSt9result_ofIFT0_iEE4typeEE4typeEEESB_iOSE_]+0x1d): undefined reference to `boost::chrono::steady_clock::now()'
direct-simple.cpp:(.text._ZN6nonius6detail16run_for_at_leastIN5boost6chrono12steady_clockENS0_8repeaterINS_3nowIS4_EEEEEENS_6timingINT_8durationENS0_13complete_typeINSt9result_ofIFT0_iEE4typeEE4typeEEESB_iOSE_[_ZN6nonius6detail16run_for_at_leastIN5boost6chrono12steady_clockENS0_8repeaterINS_3nowIS4_EEEEEENS_6timingINT_8durationENS0_13complete_typeINSt9result_ofIFT0_iEE4typeEE4typeEEESB_iOSE_]+0x41): undefined reference to `boost::chrono::steady_clock::now()'
direct-simple.cpp:(.text._ZN6nonius6detail16run_for_at_leastIN5boost6chrono12steady_clockENS0_8repeaterINS_3nowIS4_EEEEEENS_6timingINT_8durationENS0_13complete_typeINSt9result_ofIFT0_iEE4typeEE4typeEEESB_iOSE_[_ZN6nonius6detail16run_for_at_leastIN5boost6chrono12steady_clockENS0_8repeaterINS_3nowIS4_EEEEEENS_6timingINT_8durationENS0_13complete_typeINSt9result_ofIFT0_iEE4typeEE4typeEEESB_iOSE_]+0x51): undefined reference to `boost::chrono::steady_clock::now()'
direct-simple.cpp:(.text._ZN6nonius6detail16run_for_at_leastIN5boost6chrono12steady_clockENS0_8repeaterINS_3nowIS4_EEEEEENS_6timingINT_8durationENS0_13complete_typeINSt9result_ofIFT0_iEE4typeEE4typeEEESB_iOSE_[_ZN6nonius6detail16run_for_at_leastIN5boost6chrono12steady_clockENS0_8repeaterINS_3nowIS4_EEEEEENS_6timingINT_8durationENS0_13complete_typeINSt9result_ofIFT0_iEE4typeEE4typeEEESB_iOSE_]+0x5b): undefined reference to `boost::chrono::steady_clock::now()'
/tmp/direct-simple-8546e9.o:direct-simple.cpp:(.text._ZN6nonius6detail16run_for_at_leastIN5boost6chrono12steady_clockENS0_8repeaterINS_3nowIS4_EEEEEENS_6timingINT_8durationENS0_13complete_typeINSt9result_ofIFT0_iEE4typeEE4typeEEESB_iOSE_[_ZN6nonius6detail16run_for_at_leastIN5boost6chrono12steady_clockENS0_8repeaterINS_3nowIS4_EEEEEENS_6timingINT_8durationENS0_13complete_typeINSt9result_ofIFT0_iEE4typeEE4typeEEESB_iOSE_]+0x6a): more undefined references to `boost::chrono::steady_clock::now()' follow

Is this intentional? I hoped it could stay fully header-only.

Clang 3.5
Boost 1_57

Remove the Boost dependency (and make it really header only)

I evaluated several benchmark libraries for C++.
Right at the moment I could not find one benchmark library which is really header only.
For the next release, I wish you could get rid of the boost dependency and make the library a really header only library.

The test framework Catch is also header only, but does not contain a benchmark functionality.
Your library can fill this part. I also think a lot of users would like to simply add the header and start benching.

PS: what is the reason for the pthread linking on linux? Maybe you can also fix this (with std::thread?)
PPS: I also does not see an explanation for linking against boost::chrono on windows, why is this the case?

example3.c++, example4.c++ and example5.c++ failed to compile

The example1.c++ and example2.c++ compile fine, but example3.c++, example4.c++ and example5.c++ fail:

certik@redhawk:~/repos/nonius/examples(stable)$ g++ -g -std=c++11 -I../include -I$HASHSTACK/include example1.c++ -pthread 
certik@redhawk:~/repos/nonius/examples(stable)$ g++ -g -std=c++11 -I../include -I$HASHSTACK/include example2.c++ -pthread 
certik@redhawk:~/repos/nonius/examples(stable)$ g++ -g -std=c++11 -I../include -I$HASHSTACK/include example3.c++ -pthread 
/usr/lib/../lib64/crt1.o: In function `_start':
(.text+0x20): undefined reference to `main'
collect2: error: ld returned 1 exit status
certik@redhawk:~/repos/nonius/examples(stable)$ g++ -g -std=c++11 -I../include -I$HASHSTACK/include example4.c++ -pthread 
/usr/lib/../lib64/crt1.o: In function `_start':
(.text+0x20): undefined reference to `main'
collect2: error: ld returned 1 exit status
certik@redhawk:~/repos/nonius/examples(stable)$ g++ -g -std=c++11 -I../include -I$HASHSTACK/include example5.c++ -pthread 
/usr/lib/../lib64/crt1.o: In function `_start':
(.text+0x20): undefined reference to `main'
collect2: error: ld returned 1 exit status

Also I had to apply the following patch to make them compile:

diff --git a/examples/example1.c++ b/examples/example1.c++
index 01e66bc..0dda421 100644
--- a/examples/example1.c++
+++ b/examples/example1.c++
@@ -1,4 +1,4 @@
-#include <nonius.h++>
+#include <nonius/nonius.h++>

 #include <iterator>
 #include <string>
diff --git a/examples/example2.c++ b/examples/example2.c++
index 2ad5a0c..bcca1f6 100644
--- a/examples/example2.c++
+++ b/examples/example2.c++
@@ -1,4 +1,4 @@
-#include <nonius.h++>
+#include <nonius/nonius.h++>

 #include <iterator>
 #include <string>
diff --git a/examples/example3.c++ b/examples/example3.c++
index f41a51f..24b7b7a 100644
--- a/examples/example3.c++
+++ b/examples/example3.c++
@@ -1,5 +1,5 @@
 #define NONIUS_RUNNER
-#include <nonius.h++>
+#include <nonius/nonius.h++>

 NONIUS_BENCHMARK("to_string(42)", []{
     return std::to_string(42);
diff --git a/examples/example4.c++ b/examples/example4.c++
index 727cbaa..00c51c5 100644
--- a/examples/example4.c++
+++ b/examples/example4.c++
@@ -1,5 +1,5 @@
 #define NONIUS_RUNNER
-#include <nonius.h++>
+#include <nonius/nonius.h++>

 #include <list>
 #include <forward_list>
diff --git a/examples/example5.c++ b/examples/example5.c++
index 05a5482..0ac1062 100644
--- a/examples/example5.c++
+++ b/examples/example5.c++
@@ -1,5 +1,5 @@
 #define NONIUS_RUNNER
-#include <nonius.h++>
+#include <nonius/nonius.h++>

 #include <string>
 #include <vector>

Add escape and clobber functions

Chandler Carruth introduced two interesting functions to control compiler optimization in fine-grained and unintrusive manner in his talk at CppCon2015. Nonius should provide these out of the box.

Add deprecation mechanism for old API

#65 is a breaking change, so it needs a migration strategy.

Run multiple reporters

Enable the default runner to use more than one reporter at once. This allows, e.g., getting the "standard" output while collecting all samples in CSV, and/or while also generating charts.

Highcharts license concerns

Hi!

Sorry for being the legalist party pooper...

As part of my work for #36 I was looking at how the Highcharts to see how it works... When reading the code in the template distributed in Nonius, I does not say what license Highcharts has, it just claims that the license is at www.highcharts.com/license. When one visits that page, one is redirected to a shop to buy a proprietary license.

Somewhere hidden there is an option to buy get a non-commercial license, based on Creative Commons Non Commercial. By reading their FAQ: https://shop.highsoft.com/faq#Non-Commercial-0 it seems to me that the way Nonius uses it is not following their terms, or at least the way they interpret them. E.g.

Non-Commercial Redistribution

You are allowed to distribute a Highsoft software product with non-commercial packages given that you fulfill two conditions:

Emphasize to your users that Highsoft software products are not free for commercial and Governmental use. You can do this on your download page or when your users activate a Highsoft Software Product in your application.

Provide a link back to highcharts.com in the same location.

May I use your Software under the Non-Commercial License for Open Source Projects?

Although Highsoft´s Software have open source codes, our software is not licensed as an open source software and are unfortunately not compatible with any open source software license like Apache 2 or any GPL. See also Non-Commercial Redistribution

I think that the whole licensing of that library is a mess. It is impossible to know under what license was the library distributed when it was integrated in Nonius, and if those terms allowed rewording. Was this Non-Commercial clause added after-the-fact?

Since their licensing is messy, seems hostile to open-source and even unclear for commercial purposes (they make interpretations in the FAQ that are not sure are correct from understanding CC-NC), I suggest replacing this library altogether. I don't have alternative suggestions for similar libraries, but I am going to look for them now and try to make a recommendation, and maybe implement it myself.

Cheers!

integrate with biicode

This would help other projects grab this tool easily into their testing projects as a dependency.

Fix compiler warnings in MSVC 2013 64 bit

I get a lot of compiler warning while compiling the 64 bit version of nonius.
Version: 1.0.0

e.g. something like this:

int quotes = std::count(first, last, '"');

Full MSVC 2013 64 bit output:

.../nonius/reporters/csv_reporter.h++(102): warning C4244: 'initializing' : conversion from '__int64' to 'int', possible loss of data
1>.../nonius/detail/escape.h++(32): warning C4244: 'initializing' : conversion from '__int64' to 'int', possible loss of data
1>.../nonius/reporters/junit_reporter.h++(94): warning C4244: 'initializing' : conversion from '__int64' to 'int', possible loss of data
1>.../nonius/detail/argparse.h++(73): warning C4267: 'return' : conversion from 'size_t' to 'int', possible loss of data
1>          .../nonius/detail/argparse.h++(79) : see reference to function template instantiation 'int nonius::detail::get_max_width<std::_Vector_const_iterator<std::_Vector_val<std::_Simple_types<nonius::detail::option>>>,nonius::detail::<<::<lambda_ebcb31a12dfc7293f4c5c592439004ad>>(Iterator,Iterator,Projection)' being compiled
1>          with
1>          [
1>              Iterator=std::_Vector_const_iterator<std::_Vector_val<std::_Simple_types<nonius::detail::option>>>
1>  ,            Projection=nonius::detail::<<::<lambda_ebcb31a12dfc7293f4c5c592439004ad>
1>          ]
1>.../nonius/detail/stats.h++(199): warning C4244: 'initializing' : conversion from '__int64' to 'int', possible loss of data
1>          .../nonius/detail/analyse.h++(33) : see reference to function template instantiation 'nonius::detail::bootstrap_analysis nonius::detail::analyse_samples<std::_Vector_iterator<std::_Vector_val<std::_Simple_types<double>>>>(double,int,Iterator,Iterator)' being compiled
1>          with
1>          [
1>              Iterator=std::_Vector_iterator<std::_Vector_val<std::_Simple_types<double>>>
1>          ]
1>          .../nonius/go.h++(88) : see reference to function template instantiation 'nonius::sample_analysis<boost::chrono::duration<double,boost::ratio<0x01,0x03b9aca00>>> nonius::detail::analyse<boost::chrono::duration<double,boost::ratio<0x01,0x03b9aca00>>,std::_Vector_iterator<std::_Vector_val<std::_Simple_types<boost::chrono::duration<double,boost::ratio<0x01,0x03b9aca00>>>>>>(nonius::configuration,nonius::environment<boost::chrono::duration<double,boost::ratio<0x01,0x03b9aca00>>>,Iterator,Iterator)' being compiled
1>          with
1>          [
1>              Iterator=std::_Vector_iterator<std::_Vector_val<std::_Simple_types<boost::chrono::duration<double,boost::ratio<0x01,0x03b9aca00>>>>>
1>          ]
1>          .../nonius/go.h++(130) : see reference to function template instantiation 'void nonius::go<nonius::default_clock,std::_Vector_iterator<std::_Vector_val<std::_Simple_types<nonius::benchmark>>>>(nonius::configuration,Iterator,Iterator,nonius::reporter &)' being compiled
1>          with
1>          [
1>              Iterator=std::_Vector_iterator<std::_Vector_val<std::_Simple_types<nonius::benchmark>>>
1>          ]
1>          .../nonius/main.h++(145) : see reference to function template instantiation 'void nonius::go<nonius::default_clock>(nonius::configuration,nonius::benchmark_registry &,nonius::reporter_registry &)' being compiled
1>.../nonius/detail/stats.h++(80): warning C4244: 'initializing' : conversion from '__int64' to 'int', possible loss of data
1>          .../nonius/detail/estimate_clock.h++(64) : see reference to function template instantiation 'double nonius::detail::mean<std::_Vector_iterator<std::_Vector_val<std::_Simple_types<double>>>>(Iterator,Iterator)' being compiled
1>          with
1>          [
1>              Iterator=std::_Vector_iterator<std::_Vector_val<std::_Simple_types<double>>>
1>          ]
1>          .../nonius/go.h++(42) : see reference to function template instantiation 'nonius::environment_estimate<boost::chrono::duration<double,boost::ratio<0x01,0x03b9aca00>>> nonius::detail::estimate_clock_resolution<Clock>(int)' being compiled
1>          with
1>          [
1>              Clock=nonius::default_clock
1>          ]
1>          .../nonius/go.h++(73) : see reference to function template instantiation 'nonius::environment<boost::chrono::duration<double,boost::ratio<0x01,0x03b9aca00>>> nonius::detail::measure_environment<Clock>(nonius::reporter &)' being compiled
1>          with
1>          [
1>              Clock=nonius::default_clock
1>          ]
1>.../nonius/detail/stats.h++(42): warning C4244: 'initializing' : conversion from '__int64' to 'int', possible loss of data
1>          .../nonius/detail/stats.h++(58) : see reference to function template instantiation 'double nonius::detail::weighted_average_quantile<std::_Vector_iterator<std::_Vector_val<std::_Simple_types<double>>>>(int,int,Iterator,Iterator)' being compiled
1>          with
1>          [
1>              Iterator=std::_Vector_iterator<std::_Vector_val<std::_Simple_types<double>>>
1>          ]
1>          .../nonius/detail/analyse.h++(34) : see reference to function template instantiation 'nonius::outlier_classification nonius::detail::classify_outliers<std::_Vector_iterator<std::_Vector_val<std::_Simple_types<double>>>>(Iterator,Iterator)' being compiled
1>          with
1>          [
1>              Iterator=std::_Vector_iterator<std::_Vector_val<std::_Simple_types<double>>>
1>          ]
1>.../nonius/detail/stats.h++(97): warning C4244: 'initializing' : conversion from '__int64' to 'int', possible loss of data
1>          .../nonius/detail/stats.h++(208) : see reference to function template instantiation 'nonius::detail::sample nonius::detail::resample<std::mt19937,Iterator,double(__cdecl *const )(Iterator,Iterator)>(URng &,int,Iterator,Iterator,Estimator &)' being compiled
1>          with
1>          [
1>              Iterator=std::_Vector_iterator<std::_Vector_val<std::_Simple_types<double>>>
1>  ,            URng=std::mt19937
1>  ,            Estimator=double (__cdecl *const )(std::_Vector_iterator<std::_Vector_val<std::_Simple_types<double>>>,std::_Vector_iterator<std::_Vector_val<std::_Simple_types<double>>>)
1>          ]
1>          D:\Programmierung\IDE\Microsoft Visual Studio 12.0\VC\include\xmemory0(577) : while compiling class template member function 'std::_List_node<std::pair<const _Kty,_Ty>,void *> *std::allocator<_Other>::allocate(std::allocator<_Other>::size_type)'
1>          with
1>          [
1>              _Kty=std::string
1>  ,            _Ty=std::string
1>  ,            _Other=std::_List_node<std::pair<const std::string,std::string>,void *>
1>          ]
1>          D:\Programmierung\IDE\Microsoft Visual Studio 12.0\VC\include\xmemory0(847) : see reference to function template instantiation 'std::_List_node<std::pair<const _Kty,_Ty>,void *> *std::allocator<_Other>::allocate(std::allocator<_Other>::size_type)' being compiled
1>          with
1>          [
1>              _Kty=std::string
1>  ,            _Ty=std::string
1>  ,            _Other=std::_List_node<std::pair<const std::string,std::string>,void *>
1>          ]
1>          D:\Programmierung\IDE\Microsoft Visual Studio 12.0\VC\include\xmemory0(749) : see reference to class template instantiation 'std::allocator<_Other>' being compiled
1>          with
1>          [
1>              _Other=std::_List_node<std::pair<const std::string,std::string>,void *>
1>          ]
1>          D:\Programmierung\IDE\Microsoft Visual Studio 12.0\VC\include\list(484) : see reference to class template instantiation 'std::_Wrap_alloc<std::allocator<_Other>>' being compiled
1>          with
1>          [
1>              _Other=std::_List_node<std::pair<const std::string,std::string>,void *>
1>          ]
1>          D:\Programmierung\IDE\Microsoft Visual Studio 12.0\VC\include\list(690) : see reference to class template instantiation 'std::_List_base_types<_Ty,_Alloc>' being compiled
1>          with
1>          [
1>              _Ty=std::pair<const std::string,std::string>
1>  ,            _Alloc=std::allocator<std::pair<const std::string,std::string>>
1>          ]
1>          D:\Programmierung\IDE\Microsoft Visual Studio 12.0\VC\include\list(814) : see reference to class template instantiation 'std::_List_alloc<false,std::_List_base_types<_Ty,_Alloc>>' being compiled
1>          with
1>          [
1>              _Ty=std::pair<const std::string,std::string>
1>  ,            _Alloc=std::allocator<std::pair<const std::string,std::string>>
1>          ]
1>          D:\Programmierung\IDE\Microsoft Visual Studio 12.0\VC\include\list(862) : see reference to class template instantiation 'std::_List_buy<_Ty,_Alloc>' being compiled
1>          with
1>          [
1>              _Ty=std::pair<const std::string,std::string>
1>  ,            _Alloc=std::allocator<std::pair<const std::string,std::string>>
1>          ]
1>          D:\Programmierung\IDE\Microsoft Visual Studio 12.0\VC\include\xhash(269) : see reference to class template instantiation 'std::list<std::pair<const _Kty,_Ty>,std::allocator<std::pair<const _Kty,_Ty>>>' being compiled
1>          with
1>          [
1>              _Kty=std::string
1>  ,            _Ty=std::string
1>          ]
1>          D:\Programmierung\IDE\Microsoft Visual Studio 12.0\VC\include\unordered_map(81) : see reference to class template instantiation 'std::_Hash<std::_Umap_traits<_Kty,_Ty,std::_Uhash_compare<_Kty,_Hasher,_Keyeq>,_Alloc,false>>' being compiled
1>          with
1>          [
1>              _Kty=std::string
1>  ,            _Ty=std::string
1>  ,            _Hasher=std::hash<std::string>
1>  ,            _Keyeq=std::equal_to<std::string>
1>  ,            _Alloc=std::allocator<std::pair<const std::string,std::string>>
1>          ]
1>          .../nonius/detail/argparse.h++(117) : see reference to class template instantiation 'std::unordered_map<std::string,std::string,std::hash<_Kty>,std::equal_to<_Kty>,std::allocator<std::pair<const _Kty,_Ty>>>' being compiled
1>          with
1>          [
1>              _Kty=std::string
1>  ,            _Ty=std::string
1>          ]
1>.../nonius/detail/stats.h++(131): warning C4244: 'initializing' : conversion from '__int64' to 'int', possible loss of data
1>          .../nonius/detail/stats.h++(209) : see reference to function template instantiation 'nonius::estimate<double> nonius::detail::bootstrap<Iterator,double(__cdecl *const &)(Iterator,Iterator)>(double,Iterator,Iterator,const nonius::detail::sample &,Estimator)' being compiled
1>          with
1>          [
1>              Iterator=std::_Vector_iterator<std::_Vector_val<std::_Simple_types<double>>>
1>  ,            Estimator=double (__cdecl *const &)(std::_Vector_iterator<std::_Vector_val<std::_Simple_types<double>>>,std::_Vector_iterator<std::_Vector_val<std::_Simple_types<double>>>)
1>          ]
1>.../nonius/detail/stats.h++(145): warning C4267: 'initializing' : conversion from 'size_t' to 'int', possible loss of data
1>.../nonius/detail/stats.h++(114): warning C4244: 'initializing' : conversion from '__int64' to 'int', possible loss of data
1>          .../nonius/detail/stats.h++(134) : see reference to function template instantiation 'nonius::detail::sample nonius::detail::jackknife<double(__cdecl *const &)(Iterator,Iterator),Iterator>(Estimator,Iterator,Iterator)' being compiled
1>          with
1>          [
1>              Iterator=std::_Vector_iterator<std::_Vector_val<std::_Simple_types<double>>>
1>  ,            Estimator=double (__cdecl *const &)(std::_Vector_iterator<std::_Vector_val<std::_Simple_types<double>>>,std::_Vector_iterator<std::_Vector_val<std::_Simple_types<double>>>)
1>          ]
1>          .../nonius/detail/stats.h++(209) : see reference to function template instantiation 'nonius::estimate<double> nonius::detail::bootstrap<Iterator,double(__cdecl *const &)(Iterator,Iterator)>(double,Iterator,Iterator,const nonius::detail::sample &,Estimator)' being compiled
1>          with
1>          [
1>              Iterator=std::_Vector_iterator<std::_Vector_val<std::_Simple_types<double>>>
1>  ,            Estimator=double (__cdecl *const &)(std::_Vector_iterator<std::_Vector_val<std::_Simple_types<double>>>,std::_Vector_iterator<std::_Vector_val<std::_Simple_types<double>>>)
1>          ]

Additional:

Please get also rid of the C-Style casts: e.g. (int)(variable)
use instead static_cast(), That you can easily search for those casts and its also valid C++.

Fix VC++ build

The VC++ build is really broken at the moment.

libnonius / nonius Goto Github PK

nonius's People

Contributors

Stargazers

Watchers

Forkers

nonius's Issues

Recommend Projects

Recommend Topics

Recommend Org

Jobs