lfitaskforce / flatironmeeting

Meeting repo for likelihood free inference meeting

Flatiron Likelihood-Free Inference Meeting, March 18-22 2019

Meeting website here: https://lfitaskforce.github.io/FlatironMeeting/

Logo credit: @danielhey

The goal of this interdisciplinary meeting is to gather developers and users of Likelihood-Free Inference methods to share latest techniques, use cases and applications across different fields, and discuss open challenges. Part of this workshop will also be dedicated to a few hack days with the goal of seeding the development of a common probabilistic programming framework for Likelihood-Free Inference as well as collaboratively working on LFI-related hack projects.

Repository content

• Website: Source files for the meeting website
• Talks: Coming soon

Tracking of Hack Projects

We are using the GitHub issues in this repository to track and discuss hack project. Have a look the issue page or the meeting website to see existing projects.

Feel free to open a new project by using this issue template.

flatironmeeting's Issues

Setting up the landing page for the meeting

This issue is to track setting up the landing page. I went ahead and set up an empty jekyll site in the \docs folder, which is rendered at this address: https://lfitaskforce.github.io/FlatironMeeting/

[HACK] "Gold mining" from malaria simulations

"Gold mining" from malaria simulations

We want to improve LFI for an epidemiology problem (and eradicate malaria) by extracting additional information from the simulator that characterizes the latent process (cf. Johann's talk on Monday, https://arxiv.org/abs/1805.12244).

Contacts: @justinalsing @johannbrehmer
Participants: @justinalsing @johannbrehmer

Goals and deliverable

• Understand the structure of the simulator and identify stochastic pieces that depend on the parameters of interest
• Implement calculation of the joint score grad_theta log p(x, z | theta) for each run of the simulator
• Visualize the score vs summary statistics, check expectation and variance
• Discuss how this can be used in inference

Resources needed

People who like to play, ideally some experience with autodiff

Detailed description

[HACK] Porting LFI Techniques to Julia

Porting LFI Techniques to Julia

The goal is to take one or more LFI techniques implemented in a more commonly used language (e.g. Python) and directly implement it in the Julia language (i.e. without resorting to PyCall if possible).

Contacts: Danley Hsu
Participants:

Goals and deliverable

The goal is to port at least one LFI technique to Julia and release it as a public repository on Github.

Resources needed

• Some familiarity with Julia or general high-level programming concepts
• Deep familiarity with one or more LFI techniques.

Detailed description

Julia is a new programming language which attempts to take the "readability" of Python and combine it with the computational speed of languages such as C++/Fortran. As a relatively new language, statistical packages are still being developed for the language. In particular, LFI techniques are (as far as I know) practically non-existent except for a few ABC packages. I'd like to take at least one of the techniques discussed during this workshop and implement it in Julia. Not only will this provide more tools for the Julia community, but the potential boost in calculation speed could be of benefit for those who find that their existing code (e.g. in Python) runs more slowly than they'd like. In particular, Julia also has built in parallel capabilities which can be taken advantage of.

For documentation of the Julia language, see this link: Julia documentation

Drafting a schedule

This issue is to track the initial setup of the schedule. Based on our previous discussions, we were thinking of 2 days of talks and 3 days of coding, potentially with break out sessions in the middle.

[HACK] Nested sampling in PyDELFI

Nested sampling in PyDELFI

To do nested sampling over model and network parameters. Added benefit for discrete parameters because gradients are not necessary - maybe also good for multimodal distributions.

Contacts: Tom Charnock, Will Handley
Participants:

Goals and deliverable

• Small network (e.g. no hidden layer) with nested sampling which produces a posterior on n (the number of distributions) this is mixture of Gaussians
• What if we then add some hidden layers :-S

Resources needed

TENSORFLOW!!!!
An enthusiasm equal to (or greater than) Tom's enthusiasm (which is normally pretty high)

Detailed description

We would like to write a nested sampling step to be integrated into PyDELFI which would allow us to marginalise over model and network parameters. (Tom would like to write the nested sampling algorithm in TF - this maybe up for debate). This may evolve into MultiNest(erFlow)

[HACK] Global vs Local summaries

Global vs Local summaries

Compare locally optimal summaries (like those produced by IMNN, using Fisher information) with globally optimal summaries (using mutual information between summaries and parameters over the entire parameter space) for slightly challenging test cases (e.g. very non-Gaussian posterior)

Contacts: Benjamin Wandelt
Participants:

Goals and deliverable

Compare training difficulty and performance of locally and globally optimal summaries

Resources needed

A plot comparing IMNN and IMNN using mutual info for a non-Gaussian test example (heavy-tailed, or multimodal)

Detailed description

Simple NDEs for replacing MCMC with LFI

Simple NDEs for replacing MCMC with LFI

Simple NDEs (eg Gaussian with known covariance but unknown mean) for doing "known likelihood" problems with eg pydelfi.

Contacts: Justin Alsing

Resources needed

Nice intro project for someone who wants to cut their teeth with tensorflow and LFI

[HACK] Likelihood-Free Inference Toolbox

Likelihood-Free Inference Toolbox

Development of a common framework for likelihood inference based on a probabilistic programming language.

Contacts: @EiffL
Participants: @jan-matthis @hpesonen

Goals

• Gather feedback from existing project, try to answer the question: Why do people feel the need to start their own LFI package ?
• Identify the most promising backend for large scale applications (e.g. TensorFlow)
• Start the design, and possibly the implementation, of a framework meeting the requirements of practical current and future use cases.

Deliverables

• Document summarizing existing projects, their strenghts and weaknesses
• Design document for a new unified framework
• Skeleton software package with demo end-to-end LFI analysis (simulator, inference, validation)

Resources needed

• People with experience in developing, maintaining, and using a similar toolbox (e.g. ELFI) to identify use cases, limitations, and gathering feedback.
• People with state of the art methodologies, in LFI, dimensionality reduction, posterior validation, etc... who would want to see their tools accessible to, and used by, scientists in a variety of fields.
• People with experience in deploying probabilistic models at scale for scientific applications.
• People with experience in software engineering, project management, and collaborative development.

Detailed description

TBD

[HACK] Bayesian optimization for delfi

Bayesian optimization for likelihood-free inference with neural density estimators

Implement Bayesian optimization schemes for optimal acquisition of simulations for DELFI (density-estimation likelihood-free inference with NDEs)

Contacts: Justin Alsing, Tom Charnock, Stephen Feeney

Goals and deliverable

Implement Bayesian optimization schemes into eg pydelfi (or other codes doing LFI with NDEs)

Resources needed

Tensorflow, Bayesian optimization knowhow

gravitational lensing parameter estimation

Your Hack Title

Estimate the parameters of strong gravitational lenses with neural nets in a likelihood-free setting

Contacts: Yashar Hezaveh, Laurence Perreault Levasseur,
Participants: Me

Goals and deliverable

[describe your goals for the week and the deliverables you are aiming for]
One corner plot of lensing parameters with an L-free method

Resources needed

[describe the resources (software, skills, data, or just enthusiasm) needed for this project]
We have data, pre-trained feed-forward neural nets, and GPUs

Detailed description

[add additional details about the project]

We need a logo!

The current one isn't very exciting:

cc @Danielhey

[HACK] Using LFI on Exoplanet Count Data

Using LFI on Exoplanet Count Data

Applying one or more LFI techniques (not ABC) to compare with the efficiency/accuracy of results produced by ABC.

Contacts: Danley Hsu
Participants:

Goals and deliverable

Produce figures which compare the predicted occurrence rate with the true value for one or more LFI techniques and compare it the results produced by ABC.

Resources needed

• Familiarity with one or more LFI techniques which can take grid-based data

Detailed description

As part of my research, we've produced a simulator which constructs catalogs of physical planetary systems and observed properties of these systems. For occurrence rate studies, observed planets are sorted into 2D grid binned counts of observed planets which are compared with a similar grid from "true" data to gradually converge on good estimated rates using ABC. It would be useful to compare the accuracy/efficiency of ABC to other LFI techniques with sets of simulated binned counts which are created with a known "ground truth."

For papers I've authored which have made use of this scheme of ABC inference for occurrence rates, see the following two links:
Improving the Accuracy of Planet Occurrence Rates from Kepler using Approximate Bayesian Computation
Occurrence Rates of Planets orbiting FGK Stars: Combining Kepler DR25, Gaia DR2 and Bayesian Inference

[HACK] Living review for LFI

Living review for likelihood-free inference

It would be cool to have a "living review" paper for likelihood-free methods, that is kept up to date (updated say once a year or so with new methods/developments). Ideally this would act as a "practitioner's guide", and would be accessible to people doing LFI across disciplines.

Contacts: Justin Alsing, Francois Lanusse

Goals and deliverable

Is something people want? If so, sketch out how it should be structured, assign people to chapters, start writing.

Resources needed

Intimate knowledge of some corner(s) of the LFI method space, and/or the practicalities of implementation in different scenarios. Energy for writing.

[HACK] ELFI

ELFI

Improving existing features and adding new ones to the repertoire of ELFI

Contacts: @hpesonen
Participants:

Goals and deliverable

• Collect feedback on ELFI
  • What are the most important missing features and inference methods?
  • What features require improvement?
  • What types of examples are missing from the documentation/zoo?
• Start the implementations of the features.

Resources needed

• People with experience in using ELFI or other similar packages
• People wanting to develop ELFI
• People with experience in software engineering, project management, and collaborative development
• People with interesting LFI-problems

Detailed description

TBD

[HACK] NDEs for discrete (count) data

Flexible conditional density estimators for discrete data

Neural conditional density estimators such as MDNs and MAFs are great for continuous data, but often we run into discrete distributions (eg., for count/binned data). Can we build a flexible conditional density estimation framework for LFI in these situations?

Contacts: Justin Alsing
Participants:

Goals and deliverable

Develop and implement (into eg. pydelfi) a flexible conditional density estimation framework for discrete densities.

Resources needed

Background knowledge of conditional density estimation + general enthusiasm! Tensorflow/pytorch probably useful for implementation

Adding logistics information

We need some more details on the logistics page, e.g. lodging and travel info. @changhoonhahn @justinalsing you know the place, would you mind making a PR for this ?

• Website page: https://lfitaskforce.github.io/FlatironMeeting/logistics.html
• Source file: https://github.com/LFITaskForce/FlatironMeeting/blob/master/docs/logistics.md

Info to build website locally are here: https://github.com/LFITaskForce/FlatironMeeting/blob/master/CONTRIBUTING.md

Putting together registration page

This can either be done using google forms, or I'm happy to setup an heroku instance, reusing some code I made for the DESC meetings: https://github.com/LSSTDESC/meeting-registration-form

[HACK] Benchmark problems

Benchmark problems

Collect problems to test and benchmark LFI algorithms

Contacts: @jan-matthis
Participants: @johannbrehmer, @JoeriHermans, @dch216, @umbertosimola, @hpesonen, @EiffL, @OwenThomas

Goals and deliverable

• Table of existing benchmarks from the literature, figure out dimensions / features along which problems vary (cols in table)
• Discuss which features are covered by existing benchmarks, synthesize new problems
• Agree on language, framework, and interface for code and set up repo (also discuss scope: default parameters / priors? Metrics? Diagnostics?)
• Code up at least one or two simulators

Resources needed

☕️

Status

Results

• https://github.com/LFITaskForce/benchmark

[HACK] LFI on neutrino mass (cosmological sims already done!)

LFI on neutrino mass from cosmology

We have 100 cosmological models (x 10,000 map realizations each) and computed higher-order statistics (power spectrum, bispectrum, PDF, peak counts) on all of them. We would like help to apply LFI on them.

Contacts: Jia Liu
Participants: Jia Liu, Will Coulton

Goals and deliverable

Get a posterior using an (or multiple!) LFI method(s), and compare that with our (already done) Gaussian-likelihood result.

Resources needed

We will use statistics computed already from the MassiveNuS simulation: http://columbialensing.org/#massivenus

Detailed description

More like questions:

1. what do we need? mean and variance at each of the 100 models? or the full distribution instead of variance. If latter, I want to see the difference between the 2.
2. how do we scale to a different sky coverage that's different than our simulation size? (e.g. normally we multiple our covariance matrix by a number..)

[HACK] compare LFI packages and NUTS (just marginalize over all the latent parameters)

Your Hack Title

Try using NUTS on a test problem and compare to LFI package.

Contacts: @dfm
Participants: @dfm

Goals and deliverable

Resources needed

• Developers/users of LFI codes
• Toy test problem

Detailed description

[HACK] LFI flow chart

LFI flow chart

Contacts:
Participants: @changhoonhahn

Goals and deliverable

A flow chart to help future LFI users decide which method to use based on their constraints.

Resources needed

Expertise and enthusiasm in different LFI methods.

Detailed description

see above.