opencog / benchmark Goto Github PK

To follow up the @linas comment: here and to not reinvent the wheel I would propose using some ready C++ benchmarking library.

Requirements to such library:

• ability to keep run results
• ability to compare run results
• showing info on system which is used to perform test
• anti-optimization tricks

Google Benchmark (https://github.com/google/benchmark) seems to be a good candidate. Unfortunately there are no ready to install packages for benchmark libraries so it will be additional manual step in building procedure.

Other well known libraries:

• Celero (https://github.com/DigitalInBlue/Celero)
• Nonius (https://github.com/libnonius/nonius)
• Hayai (https://github.com/nickbruun/hayai)

Some review and comparision can be found here (full articles are here and here).

Python benchmarks are broken. At the moment all of benchmarks are guarded by #if HAVE_CYTHONX preprocessor condition:

Scope links are probably the most expensive link types to construct and insert in the AtomSpace. First calculating the hash value during construction is more expensive than for non scope links. Scope links like PatternLink, BindLink, etc, are even more expensive to construct so. Then inserting them to the AtomSpace also requires more computation like testing for alpha-equivalence.

query_benchmark does support scope links but mostly for querying, not construction and insertion...

All that to say that scope links really should be supported by atomspace_bm (if that doesn't get replaced by something else soon).

A key activity for the atomspace is getting Nodes by name, and Links, by outgoing set. Neither of these are measured.

File make_benchmark_graphs.py is part of opencog/benchmark/atomspace/atomspace benchmarkin.
Todo:

• move make_benchmark_graphs.py into benchmark repo
• fix atomspace/atomspace/README.md using new file location

The current atomspace benchmark only measures node and link creation, and not 'typical' data. resulting in real-life performance that is one or two orders of magnitude worse than what's measured. See for example: opencog/test-datasets#7

The benchmark needs to be updated to measure more 'typical' data.

One key activity is getting Nodes by name, and Links by outgoing set. This is not measured currently, it should be.