ofthegoats / abc-presentation

Resources for learning automatic differentiation and probabilistic programming

ProbFX Internship

## Automatic Differentiation

• Forward or Reverse Mode Automatic Differentiation: What’s the Difference? — Schrijvers et al
• Beautiful Differentiation — Elliott
• Functional Differentiation of Computer Programs — Karczmarczuk — old paper but potentially useful for background

## Probabilistic Programming

Moving towards likelihood-free programming. This means new distributions are written as models which specify how to generate samples from that distribution. More interestingly, we do not need to know the probability density function.

This means we can compose distributions/random variables with usual arithmetic functions and not worry about what happens to the density function!

See #6 for inference methods

abc-presentation's Issues

maximum likelihood estimation

generalise MLE

Approximate Bayesian Computation

Bayesian Synthetic Likelihood

Implement Distributions

• categorical
• gaussian (via Box-Muller?)
• beta
• gamma
• exponential
• geometric
• poisson

Reorganise repo

Originally intended to only contain examples of implementation for automatic differentiation.
Now, the focus is likely to be on probabilistic programming instead, with AD as an addition.
Organise the repo accordingly and suitably pedagogically.
Low priority.

Implement common probabilistic methods

• $\mathbb{P}$ measure?
  • some quadrature/integration
  • distinguish continuous from discrete
• $\mathbb{E}$ --- expectation
  • monte carlo method?
  • integration method?
• moment generating function --- $M_X(t) = \mathbb{E} \left(e^{tX} \right)$

Investigate errors in AD

e.g. getting NaNs from AD on normal density. Precision error?

Might like to use (multivariate?) normal distribution to approximate (auxiliary) distributions a la SL and differentiate on parameters. Suggests MVP as hardcoded normal distribution derivatives...

But would also be satisfying to have fully-functioning (and extensible?) AD, which might prove useful if I go on to do some kind of MLE.

Implement Likelihood-Free Inference

(Temporarily) discard use of probability densities, use instead likelihood-free methods to perform inference on intractable or otherwise "difficult" distributions.

• #7
  • Rejection Sampling
  • MCMC sampling
• #8

Implement Inference Methods

Implement some simple inference methods

for parameters

• #5
• method of moments
• linear regression

to sample from complicated distributions

• Metropolis-Hastings mcmc