Big Data vs. complex physical models - a scalable inference algorithm
A algorithm for fitting models against many data sets, giving parameter probability distributions. The key is that model evaluations are efficiently re-used between data sets, making the algorithm scale sub-linearly.
See paper for details: https://arxiv.org/abs/1707.04476
How to run
You need to install
- python-igraph
- numpy, scipy
- h5py
- progressbar
- gcc
Then run:
$ # build $ make $ # simulate data set $ python gensimple_horns.py 10000 $ # analyse $ OMP_NUM_THREADS=4 python sample.py data_widths_10000.hdf5 100 $ # simulate no-signal data set $ python gennothing.py 10000 # simulate no-signal data set $ # analyse $ OMP_NUM_THREADS=4 python sample.py data_nothing_10000.hdf5 10000
See paper draft for details.
Improving Performance
See TODO.
Implementation notes and Code organisation
- sample.py sets up everything
- Set your problem definition (parameters, model, likelihood) in sample.py
- Integrator: multi_nested_integrator.py . Calls sampler repeatedly.
- Joint Sampler: multi_nested_sampler.py . This deals with managing the graph and the queues and which live points to use for a new draw. Calls draw_constrained
- The queues (paper) are called shelves in the code.
- RadFriends: hiermetriclearn.py: Suggests new samples from live points and filters with likelihood function to return a higher point.
- clustering/: Fast C implementations for checking if a point is in the neighbourhood and computing safe distances.