Awesome Normalizing Flows

A list of awesome resources for understanding and applying normalizing flows (NF): a relatively simple yet powerful new tool in statistics for constructing expressive probability distributions from simple base distributions using a chain (flow) of trainable smooth bijective transformations (diffeomorphisms).

^{Figure inspired by Lilian Weng. Created in TikZ. View source.}

📝 Publications
1. 🛠️ Applications
📺 Videos
📦 Packages
🧑‍💻 Code
🌐 Blog Posts
🎉 Open to Suggestions!

📝 Publications

Apr 1, 2011 - Iterative Gaussianization: from ICA to Random Rotations by Laparra et. al.

Normalizing flows in the form of Gaussianization in an iterative format. Also shows connections to information theory.
Oct 30, 2014 - Non-linear Independent Components Estimation by Laurent Dinh, David Krueger, Yoshua Bengio.

Introduces the additive coupling layer (NICE) and shows how to use it for image generation and inpainting.
Feb 12, 2015 - Masked Autoencoder for Distribution Estimation by Mathieu Germain, Karol Gregor, Iain Murray, Hugo Larochelle.

Introduces MADE, a feed-forward network that uses carefully constructed binary masks on its weights to control the precise flow of information through the network. The masks ensure that each output unit receives signals only from input units that come before it in some arbitrary order. Yet all outputs can be computed in a single pass.
A popular and efficient method to bestow flows with autoregressivity is to construct them from MADE nets.

^{Figure created in TikZ. View source.}
May 21, 2015 - Variational Inference with Normalizing Flows by Danilo Rezende, Shakir Mohamed.

They show how to go beyond mean-field variational inference by using flows to increase the flexibility of the variational family.
May 27, 2016 - Density estimation using Real NVP by Laurent Dinh, Jascha Sohl-Dickstein, Samy Bengio.

They introduce the affine coupling layer (RNVP), a major improvement in terms of flexibility over the additive coupling layer (NICE) with unit Jacobian while keeping a single-pass forward and inverse transformation for fast sampling and density estimation, respectively.
Jun 15, 2016 - Improving Variational Inference with Inverse Autoregressive Flow by Diederik Kingma et al.
Mar 6, 2017 - Multiplicative Normalizing Flows for Variational Bayesian Neural Networks by Christos Louizos, Max Welling.

They introduce a new type of variational Bayesian neural network that uses flows to generate auxiliary random variables which boost the flexibility of the variational family by multiplying the means of a fully-factorized Gaussian posterior over network parameters. This turns the usual diagonal covariance Gaussian into something that allows for multimodality and non-linear dependencies between network parameters.
May 19, 2017 - Masked Autoregressive Flow for Density Estimation by George Papamakarios, Theo Pavlakou, Iain Murray.

Introduces MAF, a stack of autoregressive models forming a normalizing flow suitable for fast density estimation but slow at sampling. Analogous to Inverse Autoregressive Flow (IAF) except the forward and inverse passes are exchanged. Generalization of RNVP.
Mar 15, 2018 - Sylvester Normalizing Flow for Variational Inference by Rianne van den Berg, Leonard Hasenclever, Jakub M. Tomczak, Max Welling.

Introduces Sylvester normalizing flows which remove the single-unit bottleneck from planar flows for increased flexibility in the variational posterior.
April 3, 2018 - Neural Autoregressive Flows by Chin-Wei Huang, David Krueger, Alexandre Lacoste, Aaron Courville.

Unifies and generalize autoregressive and normalizing flow approaches, replacing the (conditionally) affine univariate transformations of MAF/IAF with a more general class of invertible univariate transformations expressed as monotonic neural networks. Also demonstrates that the proposed neural autoregressive flows (NAF) are universal approximators for continuous probability distributions. (Author's Code)
July 3, 2018 - Deep Density Destructors by Inouye & Ravikumar

Normalizing flows but from an iterative perspective. Features a Tree-based density estimator.
Jul 9, 2018 - Glow: Generative Flow with Invertible 1x1 Convolutions by Kingma, Dhariwal.

They show that flows using invertible 1x1 convolution achieve high likelihood on standard generative benchmarks and can efficiently synthesize realistic-looking, large images.
Oct 2, 2018 - FFJORD: Free-form Continuous Dynamics for Scalable Reversible Generative Models by Grathwohl & Chen et. al.

Uses Neural ODEs as a solver to produce continuous-time normalizing flows (CNF)
Nov 6, 2018 - FloWaveNet : A Generative Flow for Raw Audio by Kim et. al.

A flow-based generative model for raw audo synthesis. (Author's Code)
Apr 9, 2019 - Block Neural Autoregressive Flow - De Cao et. al.
May 17, 2019 - Integer Discrete Flows and Lossless Compression by Hoogeboom et. al.

A normalizing flow to be used for ordinal discrete data. They introduce a flexible transformation layer called integer discrete coupling.
May 30, 2019 - Graph Normalizing Flows by Jenny Liu et al. A new, reversible graph network for prediction and generation.

They perform similarly to message passing neural networks on supervised tasks, but at significantly reduced memory use, allowing them to scale to larger graphs. Combined with a novel graph auto-encoder for unsupervised learning, graph normalizing flows are a generative model for graph structures.
Jul 21, 2019 - Noise Regularization for Conditional Density Estimation by Rothfuss et. al.

Normalizing flows for conditional density estimation. This paper proposes noise regularization to reduce overfitting. (Blog | )
Aug 25, 2019 - Normalizing Flows: An Introduction and Review of Current Methods by Kobyzev et al.

Another very thorough and very readable review article going through the basics of NFs as well as some of the state-of-the-art. Also highly recommended.
Jun 10, 2019 - Neural Spline Flows by Conor Durkan et. al.

Uses monotonic ration splines as a coupling layer. This is currently one of the state of the art.
Dec 5, 2019 - Normalizing Flows for Probabilistic Modeling and Inference by Papamakarios et al.

A thorough and very readable review article by some of the guys at DeepMind involved in the development of flows. Highly recommended.
Jan 15, 2020 - Invertible Generative Modeling using Linear Rational Splines by Dolatabadi et. al.

A successor to the Neural spline flows which features an easy-to-compute inverse.
Jan 17, 2020 - Training Normalizing Flows with the Information Bottleneck for Competitive Generative Classification by Ardizzone et. al.

They introduce a class of conditional normalizing flows with an information bottleneck objective. (Author's Code)
Feb 16, 2020 - Stochastic Normalizing Flows by Hao Wu, Jonas Köhler, Frank Noé.

Introduces SNF, an arbitrary sequence of deterministic invertible functions (the flow) and stochastic processes such as MCMC or Langevin Dynamics. The aim is to increase expressiveness of the chosen deterministic invertible function, while the trainable flow improves sampling efficiency over pure MCMC (Tweet).
Feb 21, 2020 - Stochastic Normalizing Flows by Liam Hodgkinson, Chris van der Heide, Fred Roosta, Michael W. Mahoney.

Name clash for a very different technique from the above SNF: an extension of continuous normalizing flows using stochastic differential equations (SDE). Treats Brownian motion in the SDE as a latent variable and approximates it by a flow. Aims to enable efficient training of neural SDEs which can be used for constructing efficient Markov chains.
Feb 24, 2020 - Modeling Continuous Stochastic Processes with Dynamic Normalizing Flows by Deng et. al.

They propose a normalizing flow using differential deformation of the Wiener process. Applied to time series. (Tweet)
Feb 27, 2020 - Gradient Boosted Normalizing Flows by Giaquinto & Banerjee

Augment traditional normalizing flows with gradient boosting. They show that training multiple models can achieve good results and it's not necessary to have more complex distributions. (Author's Code)
Mar 4, 2020 - Gaussianization Flows by Meng et. al.

Uses a repeated composition of trainable kernel layers and orthogonal transformations. Very competitive versus some of the SOTA like Real-NVP, Glow and FFJORD. (Author's Code)
Mar 31, 2020 - Flows for simultaneous manifold learning and density estimation by Brehmer & Cranmer.

Normalizing flows that learn the data manifold and probability density function on that manifold. (Tweet | Author's Code)
April 8, 2020 - Normalizing Flows with Multi-Scale Autoregressive Priors by Mahajan & Bhattacharyya et. al.

Improves the representational power of flow-based models by introducing channel-wise dependencies in their latent space through multi-scale autoregressive priors (mAR) (Author's Code)
Jun 3, 2020 - Equivariant Flows: exact likelihood generative learning for symmetric densities by Jonas Köhler, Leon Klein, Frank Noé.

Shows that distributions generated by equivariant NFs faithfully reproduce symmetries in the underlying density. Proposes building blocks for flows which preserve typical symmetries in physical/chemical many-body systems. Shows that symmetry-preserving flows can provide better generalization and sampling efficiency.
Jun 15, 2020 - Why Normalizing Flows Fail to Detect Out-of-Distribution Data by Kirichenko et. al.

This study how traditional normalizing flow models can suffer from out-of-distribution data. They offer a solution to combat this issue by modifying the coupling layers. (Tweet | Author's Code)
July 6, 2020 - SurVAE Flows: Surjections to Bridge the Gap between VAEs and Flows by Didrik Nielsen et. al.

They present a generalized framework that encompasses both Flows (deterministic maps) and VAEs (stochastic maps). By seeing deterministic maps x = f(z) as limiting cases of stochastic maps x ~ p(x|z), the ELBO is reinterpreted as a change of variables formula for the stochastic maps. Moreover, they present a few examples of surjective layers using stochastic maps, which can be composed together with flow layers. (Video | Author's Code)
July 15, 2020 - AdvFlow: Inconspicuous Black-box Adversarial Attacks using Normalizing Flows by Dolatabadi et. al.

An adversarial attack method on image classifiers that use normalizing flows. (Author's Code)
Sept 21, 2020 - Haar Wavelet based Block Autoregressive Flows for Trajectories by Bhattacharyya et. al.

Introduce a Haar wavelet-based block autoregressive model.

🛠️ Applications

Aug 14, 2018 - Analyzing Inverse Problems with Invertible Neural Networks by Ardizzone et. al.

Normalizing flows for inverse problems.
Mar 9, 2019 - NeuTra-lizing Bad Geometry in Hamiltonian Monte Carlo Using Neural Transport by Hoffman et. al.

Uses normalizing flows in conjunction with Monte Carlo estimation to have more expressive distributions and better posterior estimation.
Jun 25, 2020 - SRFlow: Learning the Super-Resolution Space with Normalizing Flow by Lugmayr et. al.

Uses normalizing flows for super-resolution.
Jul 15, 2020 - Faster Uncertainty Quantification for Inverse Problems with Conditional Normalizing Flows by Siahkoohi et. al.

Uses conditional normalizing flows for inverse problems. (Video)
Oct 13, 2020 - Targeted free energy estimation via learned mappings

Normalizing flows used to estimate free energy differences.
Nov 2, 2020 - On the Sentence Embeddings from Pre-trained Language Models by Li et al.

Proposes to use flows to transform anisotropic sentence embedding distributions from BERT to a smooth and isotropic Gaussian, learned through unsupervised objective. Demonstrates performance gains over SOTA sentence embeddings on semantic textual similarity tasks. Code available at https://github.com/bohanli/BERT-flow.
Dec 6, 2020 - Normalizing Kalman Filters for Multivariate Time Series Analysis

Augments state space models with normalizing flows and thereby mitigates imprecisions stemming from idealized assumptions. Aimed at forecasting real-world data and handling varying levels of missing data. (Also available at Amazon Science.)

📺 Videos

Oct 4, 2018 - Sylvester Normalizing Flow for Variational Inference by Rianne van den Berg.

Introduces Sylvester normalizing flows which remove the single-unit bottleneck from planar flows for increased flexibility in the variational posterior.
Mar 24, 2019 - PixelCNN, Wavenet, Normalizing Flows - Santiago Pascual - UPC Barcelona 2018
Sep 24, 2019 - Graph Normalizing Flows by Jenny Liu (University of Toronto, Vector Institute).

Introduces a new graph generating model for use e.g. in drug discovery, where training on molecules that are known to bind/dissolve/etc. may help to generate novel, similarly effective molecules.
Oct 9, 2019 - A primer on normalizing flows by Laurent Dinh (Google Brain).

The first author on both the NICE and RNVP papers and one of the first in this field gives an introductory talk at "Machine Learning for Physics and the Physics Learning of, 2019".
Dec 6, 2019 - What are normalizing flows? by Ari Seff (Princeton).

A great 3blue1brown-style video explaining the basics of normalizing flows.
Flow Models by CS294-158-SP20 Deep, Unsupervised Spring Learning,, 2020 (Berkeley)

A really thorough explanation of normalizing flows. Also includes some sample code.
Nov 23, 2020 - Introduction to Normalizing Flows by Marcus Brubaker (York University, Toronto)

A great introduction to normalizing flows by one of the creators of Stan presented at ECCV 2020. The tutorial also provides an excellent review of various practical implementations.
January 16, 2021 - Normalizing Flows - Motivations, The Big Idea & Essential Foundations by Kapil Sachdeva

A comprehensive tutorial on flows explaining the challenges addressed by this class of algorithm. Provides intuition on how to address those challenges, and explains the underlying mathematics using a simple step by step approach.

📦 Packages

PyTorch Packages

Sep 7, 2018 - FrEIA by VLL Heidelberg

The Framework for Easily Invertible Architectures (FrEIA) is based on RNVP flows. Easy to setup, it allows to define complex Invertible Neural Networks (INNs) from simple invertible building blocks.
Feb 9, 2020 - nflows by Bayesiains

A suite of most of the SOTA methods using PyTorch. From an ML group in Edinburgh. They created the current SOTA spline flows. Almost as complete as you'll find from a single repo.

TensorFlow Packages

Jun 22, 2018 - TensorFlow Probability

Large first-party library that offers RNVP, MAF among other autoregressive models plus a collection of composable bijectors.

JAX Packages

Mar 9, 2020 - NuX by Information Fusion Labs (UMass)

A library that offers normalizing flows using JAX as the backend. Has some SOTA methods. They also feature a surjective flow via quantization.
Mar 23, 2020 - jax-flows by Chris Waites

Another library that has normalizing flows using JAX as the backend. Has some of the SOTA methods.
Jun 17, 2021 - pzflow by John Franklin Crenshaw

A package that focuses on probabilistic modeling of tabular data, with a focus on sampling and posterior calculation.

Julia Packages

Feb 7, 2020 - InvertibleNetworks.jl by SLIM

A Flux compatible library implementing invertible neural networks and normalizing flows using memory-efficient backpropagation. Uses manually implemented gradients to take advantage of the invertibility of building blocks, which allows for scaling to large-scale problem sizes.

🧑‍💻 Code

PyTorch Repos

Sep 1, 2018 - pytorch-flows by Ilya Kostrikov.

PyTorch implementations of density estimation algorithms: MAF, RNVP, Glow.
Dec 30, 2018 - normalizing_flows by Kamen Bliznashki.

Pytorch implementations of density estimation algorithms: BNAF, Glow, MAF, RealNVP, planar flows.
Feb 6, 2019 - pytorch_flows by acids-ircam

A great repo with some basic PyTorch implementations of normalizing flows from scratch.
Dec 9, 2019 - pytorch-normalizing-flows by Andrej Karpathy.

A Jupyter notebook with PyTorch implementations of the most commonly used flows: NICE, RNVP, MAF, Glow, NSF.
Jul 3, 2020 - Density Estimation with Neural ODEs and Density Estimation with FFJORDs by torchdyn

Example of how to use FFJORD as a continuous normalizing flow (CNF). Based on the PyTorch suite torchdyn which offers continuous neural architectures.
July 19, 2020 - Normalizing Flows - Introduction (Part 1) by pyro.ai

A tutorial about how to use the pyro-ppl library (based on PyTorch) to use Normalizing flows. They provide some SOTA methods including NSF and MAF. Parts 2 and 3 coming later.

JAX Repos

Jul 19, 2019 - Neural Transport by numpyro

Features an example of how Normalizing flows can be used to get more robust posteriors from Monte Carlo methods. Uses the numpyro library which is a PPL with JAX as the backend. The NF implementations include the basic ones like IAF and BNAF.

TensorFlow Repos

Jun 12, 2020 - Variational Inference using Normalizing Flows (VINF) by Pierre Segonne

This repository provides a hands-on TensorFlow implementation of Normalizing Flows as presented in the paper introducing the concept (D. Rezende & S. Mohamed).
Nov 2, 2020 - BERT-flow by Bohan Li

TensorFlow implementation of "On the Sentence Embeddings from Pre-trained Language Models" (EMNLP 2020)

Others

Mar 21, 2017 - 'NormFlows'

Simple didactic example using autograd, so pretty low-level.
Jul 11, 2017 - normalizing_flows_overview.ipynb by PyMC3.

A very helpful notebook showcasing how to work with flows in practice and comparing it to PyMC3's NUTS-based HMC kernel. Based on Theano.
Jun 11, 2018 - destructive-deep-learning by David Inouye
Code base for the paper Deep Density Destructors by Inouye & Ravikumar (2018). An entire suite of iterative methods including tree-based as well as Gaussianization methods which are similar to normalizing flows except they converge iteratively instead of fully parametrized. That is, they still use bijective transforms, compute the Jacobian, check the likelihood and you can still sample and get probability density estimates. The only difference is you repeat the following two steps until convergence:
1. compute one layer or block layer (e.g. Marginal Gaussianization + PCA rotation)
2. check for convergence (e.g log-likelihood using the change-of-variables formula)
Table 1 in the paper has a good comparison with traditional NFs.

🌐 Blog Posts

Jan 17, 2018 - Normalizing Flows Tutorial by Eric Jang.

Part 1: Distributions and Determinants. Part 2: Modern Normalizing Flows. Lots of great graphics.
Apr 3, 2018 - Normalizing Flows by Adam Kosiorek.
May 19, 2018 - Change of Variables: A Precursor to Normalizing Flows by Rui Shu
Oct 13, 2018 - Flow-based Deep Generative Models by Lilian Weng.
Oct 21, 2018 - Change of Variables for Normalizing Flows by Neal Jean

🎉 Open to Suggestions

See something that's missing from this list? PRs welcome!

Papers should be peer-reviewed and published in a journal. If you're unsure if a paper or resource belongs in this list, feel free to open an issue or start a discussion. This repo is meant to be a community effort. So don't hesitate to voice an opinion.

maximevandegar / awesome-normalizing-flows Goto Github PK