yuki-koyama / mathtoolbox

Mathematical tools (interpolation, dimensionality reduction, optimization, etc.) written in C++11 with Eigen

mathtoolbox

Mathematical tools (interpolation, dimensionality reduction, optimization, etc.) written in C++11 and Eigen.

Algorithms

Scattered Data Interpolation and Function Approximation

• rbf-interpolation: Radial basis function (RBF) interpolation
• gaussian-process-regression: Gaussian process regression (GPR)

Dimensionality Reduction and Low-Dimensional Embedding

• classical-mds: Classical multi-dimensional scaling (MDS)
• som: Self-organizing map (SOM)

Numerical Optimization

• backtracking-line-search: Backtracking line search
• bayesian-optimization: Bayesian optimization
• bfgs: BFGS method
• gradient-descent: Gradient descent method
• l-bfgs: Limited-memory BFGS method
• strong-wolfe-conditions-line-search: Strong Wolfe conditions line search

Linear Algebra

• log-determinant: Log-determinant
• matrix-inversion: Matrix inversion techniques

Utilities

• acquisition-functions: Acquisition functions
• constants: Constants
• data-normalization: Data normalization
• kernel-functions: Kernel functions
• probability-distributions: Probability distributions

Dependencies

Main Library

• Eigen http://eigen.tuxfamily.org/ (brew install eigen / sudo apt install libeigen3-dev)

Python Bindings

• pybind11 https://github.com/pybind/pybind11 (included as gitsubmodule)

Examples

• optimization-test-function https://github.com/yuki-koyama/optimization-test-functions (included as gitsubmodule)
• timer https://github.com/yuki-koyama/timer (included as gitsubmodule)

Use as a C++ Library

mathtoolbox uses CMake https://cmake.org/ for building source codes. This library can be built, for example, by

git clone https://github.com/yuki-koyama/mathtoolbox.git --recursive
cd mathtoolbox
mkdir build
cd build
cmake ../
make

and optionally it can be installed to the system by

make install

When the CMake parameter MATHTOOLBOX_BUILD_EXAMPLES is set ON, the example applications are also built. (The default setting is OFF.) This is done by

cmake ../ -DMATHTOOLBOX_BUILD_EXAMPLES=ON
make

When the CMake parameter MATHTOOLBOX_PYTHON_BINDINGS is set ON, the example applications are also built. (The default setting is OFF.) This is done by

cmake ../ -DMATHTOOLBOX_PYTHON_BINDINGS=ON
make

Prerequisites

macOS:

brew install eigen

Ubuntu:

sudo apt install libeigen3-dev

Use as a Python Library

pymathtoolbox is a (sub)set of Python bindings of mathtoolbox. Tested on Python 3.8 and 3.9.

It can be installed via PyPI:

pip install git+https://github.com/yuki-koyama/mathtoolbox

Prerequisites

macOS

brew install cmake eigen

Ubuntu 16.04/18.04

sudo apt install cmake libeigen3-dev

Examples

See python-examples.

Gallery

Bayesian optimization (bayesian-optimization) solves a one-dimensional optimization problem using only a small number of function-evaluation queries.

Classical multi-dimensional scaling (classical-mds) is applied to pixel RGB values of a target image to embed them into a two-dimensional space.

Self-organizing map (som) is also applicable to pixel RGB values of a target image to learn a two-dimensional color manifold.

Projects Using mathtoolbox

• SelPh [CHI 2016] https://github.com/yuki-koyama/selph
  • classical-mds
• Sequential Line Search [SIGGRAPH 2017] https://github.com/yuki-koyama/sequential-line-search
  • acquisition-functions, data-normalization, kernel-functions, log-determinant, and probability-distributions
• Sequential Gallery [SIGGRAPH 2020] https://github.com/yuki-koyama/sequential-gallery
  • acquisition-functions and probability-distributions
• elasty https://github.com/yuki-koyama/elasty
  • l-bfgs

Contributing

Bug reports, suggestions, feature requests, and PRs are highly welcomed.

Licensing

The MIT License.

mathtoolbox's Issues

Need demos

It is much better to include a demo program for each module

Add installation instruction to README.md

It is better to add installation instruction to README.md

[New Algorithm] Spectral Clustering

It would be nice if it supports spectral clustering

Add menu (or links) to each page of docs

It is better to add menu (or links) to each page of docs

Eigen dependency definition in Cmake

There seems to be a better method to include Eigen headers:
https://eigen.tuxfamily.org/dox/TopicCMakeGuide.html

Set target_include_directories in cmake

It would be useful to set target_include_directories in cmake files.

[New Algorithm] Thin Plate Spline Interpolation

It would be nice if it supports Thin Plate Spline (TPS) interpolation.

[New Algorithm] Bayesian optimization

Now, this library has GPR. This means that we can easily implement Bayesian optimization techniques based on it.

Reference:

• https://github.com/yuki-koyama/sequential-line-search
• http://koyama.xyz/project/sequential_line_search/download/sub_document.pdf

Maximization of acquisition values in Bayesian optimization seems not successful

It seems to unreasonably choose the boundaries of the search space. This needs to be fixed.

[Docs] Link to github repository

The current docs seem not to have a link to this GitHub repository.

Add explanation of regularization to the RBF Interpolation document

It is necessary to add explanations of the equations and effects of regularization of RBF Interpolation

[New Algorithm] Polynomial interpolation

It would be nice if it supports polynomial interpolation

Create independent modules for the line search methods

numerical-optimization.hpp implements two line search methods as internal functions, but these could be treated as independent modules.

[GPR] Add support of MAP estimation of hyperparameters

[Discussion] Document template (readthedocs?)

Recently, readthedocs https://readthedocs.org/ has become more and more popular. Should we use it?

Pros:

• Beautiful template (sphinx_rtd_theme https://sphinx-rtd-theme.readthedocs.io/) available

Cons:

• Need to learn reStructuredText http://www.sphinx-doc.org/en/master/usage/restructuredtext/basics.html and rewrite existing docs to it

As an alternative choice, MkDocs https://www.mkdocs.org/ may be worth investigating, which allows us to use Markdown.

[New Algorithm] Rejection sampling

[GPR] Support data normalization

As the current implementation assumes the mean function to be zero constant, it is better to support data normalization, i.e., modify y so that mean(y) = 0.

Also, it is worth discussing the use of scaling, i.e., modify y so that var(y) = 1.

[GPR] Add sample codes

Generate warnings when linking python bindings

https://travis-ci.com/yuki-koyama/mathtoolbox/jobs/262658713#L299-L300

Scanning dependencies of target pymathtoolbox
[ 40%] Building CXX object CMakeFiles/pymathtoolbox.dir/src/pybind/python-bindings.cpp.o
[ 43%] Linking CXX shared module pymathtoolbox.cpython-36m-darwin.so
ld: warning: direct access in function 'pybind11_init_pymathtoolbox(pybind11::module&)' from file '/tmp/lto.o' to global weak symbol 'mathtoolbox::optimization::BayesianOptimizer::EvaluatePoint(Eigen::Matrix<double, -1, 1, 0, -1, 1> const&) const' from file 'libmathtoolbox.a(bayesian-optimization.cpp.o)' means the weak symbol cannot be overridden at runtime. This was likely caused by different translation units being compiled with different visibility settings.
ld: warning: direct access in function 'char const* std::__1::__find_end<bool (*)(char, char), char const*, char const*>(char const*, char const*, char const*, char const*, bool (*)(char, char), std::__1::random_access_iterator_tag, std::__1::random_access_iterator_tag)' from file '/tmp/lto.o' to global weak symbol 'std::__1::char_traits<char>::eq(char, char)' from file 'libmathtoolbox.a(gaussian-process-regression.cpp.o)' means the weak symbol cannot be overridden at runtime. This was likely caused by different translation units being compiled with different visibility settings.
[ 43%] Built target pymathtoolbox

This issue should be fixed.

Remove Travis?

Now that this repository uses GitHub Actions to test on macOS and Ubuntu (16.04/18.04), it is not necessary to rely on Travis CI for testing. It is worth considering to remove the dependency to keep the workflow as simple as possible.

[GPR] Add explanations of how GPR class can be used

It may be better to call the current "metric-mds" function as "classical-mds"

In my understanding, classical MDS is a special case of metric MDS. Currently the metric-mds function performs classical MDS. To avoid confusion, it may be better to rename it.

[New Algorithm] Isomap

It would be nice if it supports Isomap

[New Algorithm] Gaussian process regression

It would be nice if it supports Gaussian process regression

[GPR] multiple start for hyperparameter tuning

todo: add support of multiple restarting strategy for hyperparameter tuning

Support pip for python?

It would be great if this library can be used in python by simply via pip install ...

ref: https://github.com/pybind/python_example
ref: https://packaging.python.org/tutorials/packaging-projects/

Projects using this library

It would be nice to add a projects using this library or something like that to README.md.
So far, this application uses this library: https://github.com/yuki-koyama/selph

Autolink is not working in docs

Autolink is not working in the documents

RBF Interpolation does not have Eigen interface

Currently the RBF Interpolation does not have Eigen-based interfaces, but this is better to be fixed.

[GPR] Add explanations of how GPR works

Add visualizations of how classical mds works

Visualizations of how classical mds works would be helpful.

The GPR demo fails when the number of data points is large

I've just found that the GPR demo fails to optimize its hyperparameters when there are many data points (e.g., more than 100). This needs to be examined.

Windows support

Currently, this library does not support Windows. I hope this issue will be resolved in the near future (but I don't have access to Windows and am not familiar with it).

[New Algorithm] Augmented Lagrangian method

Reference: https://en.wikipedia.org/wiki/Augmented_Lagrangian_method

[GPR] Add illustration of 95%-confidence interval

It is better to have a figure showing what the 95%-confidence interval indicates.

Build error (isnan was not declared...) on Ubuntu

It could not be built on Ubuntu. Here is the error message:

/root/mathtoolbox/mathtoolbox/rbf-interpolation.cpp:71:33: error: ‘isnan’ was not declared in this scope
                 if (isnan(result))

[GPR] Return (non-squared) sigma vs. squared sigma

Currently, it returns (non-squared) sigma rather than squared sigma (i.e., variance). It may be more straightforward to return variance.

Add documents to explain APIs for numerical optimization

The optimization algorithms can be called from a unified API in defined numerical-optimization.hpp but this is not documented yet.

[New Algorithm] K-means clustering

It would be nice if it supports k-means clustering

[GPR] python binding

todo: add python bindings for GPR

The GPR demo sometimes fails to find appropriate hyperparameters

The current demo uses maximum likelihood estimation for hyperparameters, but it fails sometimes. Especially, the length-scale hyperparameter in the ARD squared exponential kernel becomes unrealistically small, resulting in a "no relevance" kernel matrix.

A possible fix is to use maximum a posteriori estimation with an appropriate prior assumption for the hyperparameters.

Remove NLopt dependency?

Now that the library contains numerical optimization algorithms, it may be possible to remove the NLopt dependency. This can be future work.

[New Algorithm] Uniform Sampling

Simple uniform sampling ( http://en.cppreference.com/w/cpp/numeric/random/uniform_real_distribution ) with Eigen interface would be useful

[New Algorithm] Metropolis-Hastings

[New Algorithm] Gibbs sampling

RBF interpolation does not have consistent name conventions

RBF interpolation does not have consistent name conventions, which should be fixed.

Line search in BFGS and L-BFGS sometimes fail

I've just found that the demo of GPR sometimes fails to optimize hyperparameters using BFGS or L-BFGS in this library; it throws run-time exceptions during line search. This needs to be examined carefully.

Add support of GitHub Actions for Windows/macOS

Currently, only Ubuntu is supported.

Need documents

It is necessary to write some documentations about (1) how to use the library and (2) which equations are implemented.

The input matrix of metric-mds should be fixed

The current code of metric-mds takes a squared distance matrix as input, but this is not natural. It is more natural if it takes a distance matrix as input.