janosh / awesome-normalizing-flows

Awesome resources on normalizing flows.

Awesome Normalizing Flows

A list of awesome resources for understanding and applying normalizing flows (NF): a relatively simple yet powerful new tool in statistics for constructing expressive probability distributions from simple base distributions using a chain (flow) of trainable smooth bijective transformations (diffeomorphisms).

^{Figure inspired by Lilian Weng. Created in TikZ. View source.}

 Table of Contents

1. 📝 Publications
2. 🛠️ Applications
3. 📺 Videos
4. 📦 Packages
   1.  PyTorch Packages
   2.  TensorFlow Packages
   3.  JAX Packages
   4.  Julia Packages
5. 🧑‍💻 Repos
   1.  PyTorch Repos
   2.  JAX Repos
   3.  TensorFlow Repos
   4.  Other Repos
6. 🌐 Blog Posts
7. 🚧 Contributing

📝 Publications (57)

1. 2023-01-03 - FInC Flow: Fast and Invertible k×k Convolutions for Normalizing Flows by Kallapa, Nagar et al.
   propose a k×k convolutional layer and Deep Normalizing Flow architecture which i) has a fast parallel inversion algorithm with running time O(nk^2) (n is height and width of the input image and k is kernel size), ii) masks the minimal amount of learnable parameters in a layer. iii) gives better forward pass and sampling times comparable to other k×k convolution-based models on real-world benchmarks. We provide an implementation of the proposed parallel algorithm for sampling using our invertible convolutions on GPUs. [Code]

2. 2022-10-15 - Invertible Monotone Operators for Normalizing Flows by Ahn, Kim et al.
   This work proposes the monotone formulation to overcome the issue of the Lipschitz constants in previous ResNet-based normalizing flows using monotone operators and provides an in-depth theoretical analysis. Furthermore, this work constructs an activation function called Concatenated Pila (CPila) to improve gradient flow. The resulting model, Monotone Flows, exhibits an excellent performance on multiple density estimation benchmarks (MNIST, CIFAR-10, ImageNet32, ImageNet64). [Code]

3. 2022-08-18 - ManiFlow: Implicitly Representing Manifolds with Normalizing Flows by Postels, Danelljan et al.
   The invertibility constraint of NFs imposes limitations on data distributions that reside on lower dimensional manifolds embedded in higher dimensional space. This is often bypassed by adding noise to the data which impacts generated sample quality. This work generates samples from the original data distribution given full knowledge of perturbed distribution and noise model. They establish NFs trained on perturbed data implicitly represent the manifold in regions of maximum likelihood, then propose an optimization objective that recovers the most likely point on the manifold given a sample from the perturbed distribution.

4. 2022-06-03 - Graphical Normalizing Flows by Wehenkel, Louppe
   This work revisits coupling and autoregressive transformations as probabilistic graphical models showing they reduce to Bayesian networks with a pre-defined topology. From this new perspective, the authors propose the graphical normalizing flow, a new invertible transformation with either a prescribed or a learnable graphical structure. This model provides a promising way to inject domain knowledge into normalizing flows while preserving both the interpretability of Bayesian networks and the representation capacity of normalizing flows. [Code]

5. 2022-05-16 - Multi-scale Attention Flow for Probabilistic Time Series Forecasting by Feng, Xu et al.
   Proposes a novel non-autoregressive deep learning model, called Multi-scale Attention Normalizing Flow(MANF), where one integrates multi-scale attention and relative position information and the multivariate data distribution is represented by the conditioned normalizing flow.

6. 2022-01-14 - E(n) Equivariant Normalizing Flows by Satorras, Hoogeboom et al.
   Introduces equivariant graph neural networks into the normalizing flow framework which combine to give invertible equivariant functions. Demonstrates their flow beats prior equivariant models and allows sampling of molecular configurations with positions, atom types and charges.

7. 2021-07-03 - CInC Flow: Characterizable Invertible 3x3 Convolution by Nagar, Dufraisse et al.
   Seeks to improve expensive convolutions. They investigate the conditions for when 3x3 convolutions are invertible under which conditions (e.g. padding) and saw successful speedups. Furthermore, they developed a more expressive, invertible Quad coupling layer. [Code]

8. 2021-04-14 - Orthogonalizing Convolutional Layers with the Cayley Transform by Trockman, Kolter
   Parametrizes the multichannel convolution to be orthogonal via the Cayley transform (skew-symmetric convolutions in the Fourier domain). This enables the inverse to be computed efficiently. [Code]

9. 2021-04-14 - Improving Normalizing Flows via Better Orthogonal Parameterizations by Goliński, Lezcano-Casado et al.
   Parametrizes the 1x1 convolution via the exponential map and the Cayley map. They demonstrate an improved optimization for the Sylvester normalizing flows.

10. 2020-09-28 - Multivariate Probabilistic Time Series Forecasting via Conditioned Normalizing Flows by Rasul, Sheikh et al.
    Models the multi-variate temporal dynamics of time series via an autoregressive deep learning model, where the data distribution is represented by a conditioned normalizing flow. [OpenReview.net] [Code]

11. 2020-09-21 - Haar Wavelet based Block Autoregressive Flows for Trajectories by Bhattacharyya, Straehle et al.
    Introduce a Haar wavelet-based block autoregressive model.

12. 2020-07-15 - AdvFlow: Inconspicuous Black-box Adversarial Attacks using Normalizing Flows by Dolatabadi, Erfani et al.
    An adversarial attack method on image classifiers that use normalizing flows. [Code]

13. 2020-07-06 - SurVAE Flows: Surjections to Bridge the Gap between VAEs and Flows by Nielsen, Jaini et al.
    They present a generalized framework that encompasses both Flows (deterministic maps) and VAEs (stochastic maps). By seeing deterministic maps x = f(z) as limiting cases of stochastic maps x ~ p(x|z), the ELBO is reinterpreted as a change of variables formula for the stochastic maps. Moreover, they present a few examples of surjective layers using stochastic maps, which can be composed together with flow layers. [Video] [Code]

14. 2020-06-15 - Why Normalizing Flows Fail to Detect Out-of-Distribution Data by Kirichenko, Izmailov et al.
    This study how traditional normalizing flow models can suffer from out-of-distribution data. They offer a solution to combat this issue by modifying the coupling layers. [Tweet] [Code]

15. 2020-06-03 - Equivariant Flows: exact likelihood generative learning for symmetric densities by Köhler, Klein et al.
    Shows that distributions generated by equivariant NFs faithfully reproduce symmetries in the underlying density. Proposes building blocks for flows which preserve typical symmetries in physical/chemical many-body systems. Shows that symmetry-preserving flows can provide better generalization and sampling efficiency.

16. 2020-06-02 - The Convolution Exponential and Generalized Sylvester Flows by Hoogeboom, Satorras et al.
    Introduces exponential convolution to add the spatial dependencies in linear layers as an improvement of the 1x1 convolutions. It uses matrix exponentials to create cheap and invertible layers. They also use this new architecture to create convolutional Sylvester flows and graph convolutional exponentials. [Code]

17. 2020-05-11 - iUNets: Fully invertible U-Nets with Learnable Upand Downsampling by Etmann, Ke et al.
    Extends the classical UNet to be fully invertible by enabling invertible, orthogonal upsampling and downsampling layers. It is rather efficient so it should be able to enable stable training of deeper and larger networks.

18. 2020-04-08 - Normalizing Flows with Multi-Scale Autoregressive Priors by Mahajan, Bhattacharyya et al.
    Improves the representational power of flow-based models by introducing channel-wise dependencies in their latent space through multi-scale autoregressive priors (mAR). [Code]

19. 2020-03-31 - Flows for simultaneous manifold learning and density estimation by Brehmer, Cranmer
    Normalizing flows that learn the data manifold and probability density function on that manifold. [Tweet] [Code]

20. 2020-03-04 - Gaussianization Flows by Meng, Song et al.
    Uses a repeated composition of trainable kernel layers and orthogonal transformations. Very competitive versus some of the SOTA like Real-NVP, Glow and FFJORD. [Code]

21. 2020-02-27 - Gradient Boosted Normalizing Flows by Giaquinto, Banerjee
    Augment traditional normalizing flows with gradient boosting. They show that training multiple models can achieve good results and it's not necessary to have more complex distributions. [Code]

22. 2020-02-24 - Modeling Continuous Stochastic Processes with Dynamic Normalizing Flows by Deng, Chang et al.
    They propose a normalizing flow using differential deformation of the Wiener process. Applied to time series. [Tweet]

23. 2020-02-21 - Stochastic Normalizing Flows by Hodgkinson, Heide et al.
    Name clash for a very different technique from the above SNF: an extension of continuous normalizing flows using stochastic differential equations (SDE). Treats Brownian motion in the SDE as a latent variable and approximates it by a flow. Aims to enable efficient training of neural SDEs which can be used for constructing efficient Markov chains.

24. 2020-02-16 - Stochastic Normalizing Flows (SNF) by Wu, Köhler et al.
    Introduces SNF, an arbitrary sequence of deterministic invertible functions (the flow) and stochastic processes such as MCMC or Langevin Dynamics. The aim is to increase expressiveness of the chosen deterministic invertible function, while the trainable flow improves sampling efficiency over pure MCMC [Tweet).]

25. 2020-01-17 - Training Normalizing Flows with the Information Bottleneck for Competitive Generative Classification by Ardizzone, Mackowiak et al.
    They introduce a class of conditional normalizing flows with an information bottleneck objective. [Code]

26. 2020-01-15 - Invertible Generative Modeling using Linear Rational Splines by Dolatabadi, Erfani et al.
    A successor to the Neural spline flows which features an easy-to-compute inverse.

27. 2019-12-05 - Normalizing Flows for Probabilistic Modeling and Inference by Papamakarios, Nalisnick et al.
    A thorough and very readable review article by some of the guys at DeepMind involved in the development of flows. Highly recommended.

28. 2019-09-14 - Unconstrained Monotonic Neural Networks by Wehenkel, Louppe
    UMNN relaxes the constraints on weights and activation functions of monotonic neural networks by setting the derivative of the transformation as the output of an unconstrained neural network. The transformation itself is computed by numerical integration (Clenshaw-Curtis quadrature) of the derivative. [Code]

29. 2019-08-25 - Normalizing Flows: An Introduction and Review of Current Methods by Kobyzev, Prince et al.
    Another very thorough and very readable review article going through the basics of NFs as well as some of the state-of-the-art. Also highly recommended.

30. 2019-07-21 - Noise Regularization for Conditional Density Estimation by Rothfuss, Ferreira et al.
    Normalizing flows for conditional density estimation. This paper proposes noise regularization to reduce overfitting. [Blog]

31. 2019-07-18 - MintNet: Building Invertible Neural Networks with Masked Convolutions by Song, Meng et al.
    Creates an autoregressive-like coupling layer via masked convolutions which is fast and efficient to evaluate. [Code]

32. 2019-07-18 - Densely connected normalizing flows by Grcić, Grubišić et al.
    Creates a nested coupling structure to add more expressivity to standard coupling layers. They also utilize slicing/factorization for dimensionality reduction and Nystromer for the coupling layer conditioning network. They achieved SOTA results for normalizing flow models. [Code]

33. 2019-06-15 - Invertible Convolutional Flow by Karami, Schuurmans et al.
    Introduces convolutional layers that are circular and symmetric. The layer is invertible and cheap to evaluate. They also showcase how one can design non-linear elementwise bijectors that induce special properties via constraining the loss function. [Code]

34. 2019-06-15 - Invertible Convolutional Networks by Finzi, Izmailov et al.
    Showcases how standard convolutional layers can be made invertible via Fourier transformations. They also introduce better activations which might be better suited to normalizing flows, e.g. SneakyRELU

35. 2019-06-10 - Neural Spline Flows by Durkan, Bekasov et al.
    Uses monotonic ration splines as a coupling layer. This is currently one of the state of the art.

36. 2019-05-30 - Graph Normalizing Flows by Liu, Kumar et al.
    A new, reversible graph network for prediction and generation. They perform similarly to message passing neural networks on supervised tasks, but at significantly reduced memory use, allowing them to scale to larger graphs. Combined with a novel graph auto-encoder for unsupervised learning, graph normalizing flows are a generative model for graph structures.

37. 2019-05-24 - Fast Flow Reconstruction via Robust Invertible n x n Convolution by Truong, Luu et al.
    Seeks to overcome the limitation of 1x1 convolutions and proposes invertible nxn convolutions via a clever convolutional affine function.

38. 2019-05-17 - Integer Discrete Flows and Lossless Compression by Hoogeboom, Peters et al.
    A normalizing flow to be used for ordinal discrete data. They introduce a flexible transformation layer called integer discrete coupling.

39. 2019-04-09 - Block Neural Autoregressive Flow) by Cao, Titov et al.
    Introduces (B-NAF), a more efficient probability density approximator. Claims to be competitive with other flows across datasets while using orders of magnitude fewer parameters.

40. 2019-04-09 - Block Neural Autoregressive Flow by Wehenkel, Louppe
    As an alternative to hand-crafted bijections, Huang et al. (2018) proposed NAF, a universal approximator for density functions. Their flow is a neural net whose parameters are predicted by another NN. The latter grows quadratically with the size of the former which is inefficient. We propose block neural autoregressive flow (B-NAF), a much more compact universal approximator of density functions, where we model a bijection directly using a single feed-forward network. Invertibility is ensured by carefully designing affine transformations with block matrices that make the flow autoregressive and monotone. We compare B-NAF to NAF and show our flow is competitive across datasets while using orders of magnitude fewer parameters. [Code]

41. 2019-02-19 - MaCow: Masked Convolutional Generative Flow by Ma, Kong et al.
    Introduces a masked convolutional generative flow (MaCow) layer using a small kernel to capture local connectivity. They showed some improvement over the GLOW model while being fast and stable.

42. 2019-01-30 - Emerging Convolutions for Generative Normalizing Flows by Hoogeboom, Berg et al.
    Introduces autoregressive-like convolutional layers that operate on the channel and spatial axes. This improved upon the performance of image datasets compared to the standard 1x1 Convolutions. The trade-off is that the inverse operator is quite expensive however the authors provide a fast C++ implementation. [Code]

43. 2018-11-06 - FloWaveNet : A Generative Flow for Raw Audio by Kim, Lee et al.
    A flow-based generative model for raw audo synthesis. [Code]

44. 2018-10-02 - FFJORD: Free-form Continuous Dynamics for Scalable Reversible Generative Models by Grathwohl, Chen et al.
    Uses Neural ODEs as a solver to produce continuous-time normalizing flows (CNF).

45. 2018-07-09 - Glow: Generative Flow with Invertible 1x1 Convolutions by Kingma, Dhariwal
    They show that flows using invertible 1x1 convolution achieve high likelihood on standard generative benchmarks and can efficiently synthesize realistic-looking, large images.

46. 2018-07-03 - Deep Density Destructors by Inouye, Ravikumar
    Normalizing flows but from an iterative perspective. Features a Tree-based density estimator.

47. 2018-04-03 - Neural Autoregressive Flows by Huang, Krueger et al.
    Unifies and generalize autoregressive and normalizing flow approaches, replacing the (conditionally) affine univariate transformations of MAF/IAF with a more general class of invertible univariate transformations expressed as monotonic neural networks. Also demonstrates that the proposed neural autoregressive flows (NAF) are universal approximators for continuous probability distributions. [Code]

48. 2018-03-15 - Sylvester Normalizing Flow for Variational Inference by Berg, Hasenclever et al.
    Introduces Sylvester normalizing flows which remove the single-unit bottleneck from planar flows for increased flexibility in the variational posterior.

49. 2017-11-17 - Convolutional Normalizing Flows by Zheng, Yang et al.
    Introduces normalizing flows that take advantage of convolutions (based on convolution over the dimensions of random input vector) to improve the posterior in the variational inference framework. This also reduced the number of parameters due to the convolutions.

50. 2017-05-19 - Masked Autoregressive Flow for Density Estimation by Papamakarios, Pavlakou et al.
    Introduces MAF, a stack of autoregressive models forming a normalizing flow suitable for fast density estimation but slow at sampling. Analogous to Inverse Autoregressive Flow (IAF) except the forward and inverse passes are exchanged. Generalization of RNVP.

    

51. 2017-03-06 - Multiplicative Normalizing Flows for Variational Bayesian Neural Networks by Louizos, Welling
    They introduce a new type of variational Bayesian neural network that uses flows to generate auxiliary random variables which boost the flexibility of the variational family by multiplying the means of a fully-factorized Gaussian posterior over network parameters. This turns the usual diagonal covariance Gaussian into something that allows for multimodality and non-linear dependencies between network parameters.

52. 2016-06-15 - Improving Variational Inference with Inverse Autoregressive Flow by Kingma, Salimans et al.
    Introduces inverse autoregressive flow (IAF), a new type of flow which scales well to high-dimensional latent spaces. [Code]

53. 2016-05-27 - Density estimation using Real NVP by Dinh, Sohl-Dickstein et al.
    They introduce the affine coupling layer (RNVP), a major improvement in terms of flexibility over the additive coupling layer (NICE) with unit Jacobian while keeping a single-pass forward and inverse transformation for fast sampling and density estimation, respectively.

    

54. 2015-05-21 - Variational Inference with Normalizing Flows by Rezende, Mohamed
    They show how to go beyond mean-field variational inference by using flows to increase the flexibility of the variational family.

55. 2015-02-12 - Masked Autoencoder for Distribution Estimation by Germain, Gregor et al.
    Introduces MADE, a feed-forward network that uses carefully constructed binary masks on its weights to control the precise flow of information through the network. The masks ensure that each output unit receives signals only from input units that come before it in some arbitrary order. Yet all outputs can be computed in a single pass.

    A popular and efficient way to make flows autoregressive is to construct them from MADE nets.

    

56. 2014-10-30 - Non-linear Independent Components Estimation by Dinh, Krueger et al.
    Introduces the additive coupling layer (NICE) and shows how to use it for image generation and inpainting.

57. 2011-04-01 - Iterative Gaussianization: from ICA to Random Rotations by Laparra, Camps-Valls et al.
    Normalizing flows in the form of Gaussianization in an iterative format. Also shows connections to information theory.

🛠️ Applications (8)

1. 2020-12-06 - Normalizing Kalman Filters for Multivariate Time Series Analysis by Bézenac, Rangapuram et al.
   Augments state space models with normalizing flows and thereby mitigates imprecisions stemming from idealized assumptions. Aimed at forecasting real-world data and handling varying levels of missing data. (Also available at Amazon Science.)

2. 2020-11-02 - On the Sentence Embeddings from Pre-trained Language Models by Li, Zhou et al.
   Proposes to use flows to transform anisotropic sentence embedding distributions from BERT to a smooth and isotropic Gaussian, learned through unsupervised objective. Demonstrates performance gains over SOTA sentence embeddings on semantic textual similarity tasks. Code available at https://github.com/bohanli/BERT-flow.

3. 2020-10-13 - Targeted free energy estimation via learned mappings by Wirnsberger, Ballard et al.
   Normalizing flows used to estimate free energy differences.

4. 2020-07-15 - Faster Uncertainty Quantification for Inverse Problems with Conditional Normalizing Flows by Siahkoohi, Rizzuti et al.
   Uses conditional normalizing flows for inverse problems. [Video]

5. 2020-06-25 - SRFlow: Learning the Super-Resolution Space with Normalizing Flow by Lugmayr, Danelljan et al.
   Uses normalizing flows for super-resolution.

6. 2019-03-09 - NeuTra-lizing Bad Geometry in Hamiltonian Monte Carlo Using Neural Transport by Hoffman, Sountsov et al.
   Uses normalizing flows in conjunction with Monte Carlo estimation to have more expressive distributions and better posterior estimation.

7. 2018-08-14 - Analyzing Inverse Problems with Invertible Neural Networks by Ardizzone, Kruse et al.
   Normalizing flows for inverse problems.

8. 2018-04-09 - Latent Space Policies for Hierarchical Reinforcement Learning by Haarnoja, Hartikainen et al.
   Uses normalizing flows, specifically RealNVPs, as policies for reinforcement learning and also applies them for the hierarchical reinforcement learning setting.

📺 Videos (8)

1. 2021-01-16 - Normalizing Flows - Motivations, The Big Idea & Essential Foundations by Kapil Sachdeva
   A comprehensive tutorial on flows explaining the challenges addressed by this class of algorithm. Provides intuition on how to address those challenges, and explains the underlying mathematics using a simple step by step approach.

2. 2020-12-07 - Normalizing Flows by Marc Deisenroth
   Part of a NeurIPS 2020 tutorial series titled "There and Back Again: A Tale of Slopes and Expectations". Link to full series.

3. 2020-11-23 - Introduction to Normalizing Flows by Marcus Brubaker
   A great introduction to normalizing flows by one of the creators of Stan presented at ECCV 2020. The tutorial also provides an excellent review of various practical implementations.

4. 2020-02-06 - Flow Models by Pieter Abbeel
   A really thorough explanation of normalizing flows. Also includes some sample code.

5. 2019-12-06 - What are normalizing flows? by Ari Seff
   A great 3blue1brown-style video explaining the basics of normalizing flows.

6. 2019-10-09 - A primer on normalizing flows by Laurent Dinh
   The first author on both the NICE and RNVP papers and one of the first in this field gives an introductory talk at "Machine Learning for Physics and the Physics Learning of, 2019".

7. 2019-09-24 - Graph Normalizing Flows by Jenny Liu
   Introduces a new graph generating model for use e.g. in drug discovery, where training on molecules that are known to bind/dissolve/etc. may help to generate novel, similarly effective molecules.

8. 2018-10-04 - Sylvester Normalizing Flow for Variational Inference by Rianne van den Berg
   Introduces Sylvester normalizing flows which remove the single-unit bottleneck from planar flows for increased flexibility in the variational posterior.

📦 Packages (13)

 PyTorch Packages

1. 2022-05-21 - Zuko by François Rozet  
   Zuko is a Python package that implements normalizing flows in PyTorch. It relies heavily on PyTorch's built-in distributions and transformations, which makes the implementation concise, easy to understand and extend. The API is fully documented with references to the original papers.

Zuko is used in LAMPE to enable Likelihood-free AMortized Posterior Estimation with PyTorch.

1. 2021-01-25 - Jammy Flows by Thorsten Glüsenkamp  
   A package that models joint (conditional) PDFs on tensor products of manifolds (Euclidean, sphere, interval, simplex) - like inverse autoregressive flows, but connects manifolds, models conditional PDFs, and allows for arbitrary couplings instead of affine ones. Includes a few SOTA flows like Gaussianization flows.

2. 2020-12-07 - flowtorch by Facebook / Meta  
   FlowTorch Docs is a PyTorch library for learning and sampling from complex probability distributions using a class of methods called Normalizing Flows.

3. 2020-02-09 - nflows by Bayesiains  
   A suite of most of the SOTA methods using PyTorch. From an ML group in Edinburgh. They created the current SOTA spline flows. Almost as complete as you'll find from a single repo.

4. 2020-01-28 - normflows by Vincent Stimper  
   The library provides most of the common normalizing flow architectures. It also includes stochastic layers, flows on tori and spheres, and other tools that are particularly useful for applications to the physical sciences.

5. 2018-09-07 - FrEIA by VLL Heidelberg  
   The Framework for Easily Invertible Architectures (FrEIA) is based on RNVP flows. Easy to setup, it allows to define complex Invertible Neural Networks (INNs) from simple invertible building blocks.

 TensorFlow Packages

1. 2018-06-22 - TensorFlow Probability by Google  
   Large first-party library that offers RNVP, MAF among other autoregressive models plus a collection of composable bijectors.

 JAX Packages

1. 2021-06-17 - pzflow by John Franklin Crenshaw  
   A package that focuses on probabilistic modeling of tabular data, with a focus on sampling and posterior calculation.

2. 2021-04-12 - Distrax by DeepMind  
   Distrax is a lightweight library of probability distributions and bijectors. It acts as a JAX-native re-implementation of a subset of TensorFlow Probability (TFP), with some new features and emphasis on extensibility.

3. 2020-03-23 - jax-flows by Chris Waites  
   Another library that has normalizing flows using JAX as the backend. Has some of the SOTA methods.

4. 2020-03-09 - NuX by Information Fusion Labs (UMass)  
   A library that offers normalizing flows using JAX as the backend. Has some SOTA methods. They also feature a surjective flow via quantization.

 Julia Packages

1. 2021-11-07 - ContinuousNormalizingFlows.jl by Hossein Pourbozorg  
   Implementations of Infinitesimal Continuous Normalizing Flows Algorithms in Julia. [Docs]

2. 2020-02-07 - InvertibleNetworks.jl by SLIM  
   A Flux compatible library implementing invertible neural networks and normalizing flows using memory-efficient backpropagation. Uses manually implemented gradients to take advantage of the invertibility of building blocks, which allows for scaling to large-scale problem sizes.

🧑‍💻 Repos (18)

 PyTorch Repos

1. 2021-09-27 - DeeProb-kit by Lorenzo Loconte  
   A general-purpose Python library providing a collection of deep probabilistic models (DPMs) which are easy to use and extend. Implements flows such as MAF, RealNVP and NICE.

2. 2021-08-21 - NICE: Non-linear Independent Components Estimation by Maxime Vandegar  
   PyTorch implementation that reproduces results from the paper NICE in about 100 lines of code.

3. 2020-07-19 - Normalizing Flows - Introduction (Part 1) by pyro.ai
   A tutorial about how to use the pyro-ppl library (based on PyTorch) to use Normalizing flows. They provide some SOTA methods including NSF and MAF. Parts 2 and 3 coming later.

4. 2020-07-03 - Density Estimation with Neural ODEs and Density Estimation with FFJORDs by torchdyn
   Example of how to use FFJORD as a continuous normalizing flow (CNF). Based on the PyTorch suite torchdyn which offers continuous neural architectures.

5. 2020-05-26 - StyleFlow by Rameen Abdal  
   Attribute-conditioned Exploration of StyleGAN-generated Images using Conditional Continuous Normalizing Flows. [Docs]

6. 2020-02-04 - Graphical Normalizing Flows by Antoine Wehenkel  
   Official implementation of "Graphical Normalizing Flows" and the experiments presented in the paper.

7. 2019-12-09 - pytorch-normalizing-flows by Andrej Karpathy  
   A Jupyter notebook with PyTorch implementations of the most commonly used flows: NICE, RNVP, MAF, Glow, NSF.

8. 2019-09-19 - Unconstrained Monotonic Neural Networks (UMNN) by Antoine Wehenkel  
   Official implementation of "Unconstrained Monotonic Neural Networks" and the experiments presented in the paper.

9. 2019-02-06 - pytorch_flows by acids-ircam  
   A great repo with some basic PyTorch implementations of normalizing flows from scratch.

10. 2018-12-30 - normalizing_flows by Kamen Bliznashki  
    Pytorch implementations of density estimation algorithms: BNAF, Glow, MAF, RealNVP, planar flows.

11. 2018-09-01 - pytorch-flows by Ilya Kostrikov  
    PyTorch implementations of density estimation algorithms: MAF, RNVP, Glow.

 TensorFlow Repos

1. 2020-11-02 - Variational Inference using Normalizing Flows (VINF) by Pierre Segonne  
   This repository provides a hands-on TensorFlow implementation of Normalizing Flows as presented in the paper introducing the concept (D. Rezende & S. Mohamed).

2. 2020-01-29 - Normalizing Flows by Lukas Rinder  
   Implementation of normalizing flows (Planar Flow, Radial Flow, Real NVP, Masked Autoregressive Flow (MAF), Inverse Autoregressive Flow (IAF), Neural Spline Flow) in TensorFlow 2 including a small tutorial.

3. 2019-07-19 - BERT-flow by Bohan Li  
   TensorFlow implementation of "On the Sentence Embeddings from Pre-trained Language Models" (EMNLP 2020).

 JAX Repos

1. 2020-06-12 - Neural Transport by numpyro
   Features an example of how Normalizing flows can be used to get more robust posteriors from Monte Carlo methods. Uses the numpyro library which is a PPL with JAX as the backend. The NF implementations include the basic ones like IAF and BNAF.

 Other Repos

1. 2018-06-11 - Destructive Deep Learning (ddl) by David Inouye  
   Code base for the paper Deep Density Destructors by Inouye & Ravikumar (2018). An entire suite of iterative methods including tree-based as well as Gaussianization methods which are similar to normalizing flows except they converge iteratively instead of fully parametrized. That is, they still use bijective transforms, compute the Jacobian, check the likelihood and you can still sample and get probability density estimates. The only difference is you repeat the following two steps until convergence:

2. compute one layer or block layer (e.g. Marginal Gaussianization + PCA rotation)

3. check for convergence (e.g log-likelihood using the change-of-variables formula)

Table 1 in the paper has a good comparison with traditional NFs.

1. 2017-07-11 - Normalizing Flows Overview by PyMC3
   A very helpful notebook showcasing how to work with flows in practice and comparing it to PyMC3's NUTS-based HMC kernel. Based on Theano.

2. 2017-03-21 - NormFlows by Andy Miller  
   Simple didactic example using autograd, so pretty low-level.

🌐 Blog Posts (5)

1. 2020-08-19 - Chapter on flows from the book 'Deep Learning for Molecules and Materials' by Andrew White
   A nice introduction starting with the change of variables formula (aka flow equation), going on to cover some common bijectors and finishing with a code example showing how to fit the double-moon distribution with TensorFlow Probability.

2. 2018-10-21 - Change of Variables for Normalizing Flows by Neal Jean
   Short and simple explanation of change of variables theorem i.t.o. probability mass conservation.

3. 2018-10-13 - Flow-based Deep Generative Models by Lilian Weng
   Covers change of variables, NICE, RNVP, MADE, Glow, MAF, IAF, WaveNet, PixelRNN.

4. 2018-04-03 - Normalizing Flows by Adam Kosiorek
   Introduction to flows covering change of variables, planar flow, radial flow, RNVP and autoregressive flows like MAF, IAF and Parallel WaveNet.

5. 2018-01-17 - Normalizing Flows Tutorial by Eric Jang
   Part 1: Distributions and Determinants. Part 2: Modern Normalizing Flows. Lots of great graphics.

🚧 Contributing

See something that's missing from this list? PRs welcome! A good place to find new items for the Repos section is the Normalizing Flows topic on GitHub.

Note: Don't edit the readme directly (it's auto-generated). Add your submission to the appropriate data/*.yml file.

Papers should be peer-reviewed and published in a journal. If you're unsure if a paper or resource belongs in this list, feel free to open an issue or start a discussion. This repo is meant to be a community effort. Don't hesitate to voice an opinion.

awesome-normalizing-flows's Issues

Add SurVAE Flows

Hi, thanks for the comprehensive list and nice summary of materials. I would like to recommend SurVAE Flows to the list which I had an interesting read.

They present a generalized framework (SurVAE Flows) which encompasses Flows (deterministic maps) and VAEs (stochastic maps). By seeing a deterministic map (x=f(z)) as a limiting case of a stochastic map (x~p(x|z)), the ELBO can be reinterpreted as a change of variables formula for the stochastic maps. Moreover, stochastic maps are able to model surjections, which might be useful in incorporating bottleneck architectures to Flows. They also give few examples of surjective layers, which can be composed together with Flow layers.

LAMPE: a PyTorch package for posterior estimation which implements normalizing flows

Hello 👋,

TLDR. The lampe package implements normalizing flows with PyTorch. I believe this is relevant for this collection. I hope you like it!

I'm a researcher interested in simulation-based inference and posterior estimation. I have written a low-level library for amortized posterior estimation called lampe. Initially, LAMPE was relying on nflows for its normalizing flows, but it quickly became a limitation. I was not happy with some of nflows design choices. For instance, it is only possible to sample or evaluate batches and most operators do not support broadcasting. It is also not possible to use other networks than the built-in ones. I considered contributing to nflows, but it seems the package is not actively developed anymore.

So I decided to implement my own normalizing flows within LAMPE. The goal was to rely as much as possible onto the already existing distributions and transformations of PyTorch. Unfortunately, PyTorch distributions and transforms are not modules, meaning that they don't implement a forward method, you cannot send the parameters to GPU with .to('cuda') or even get their parameters with .parameters(). To solve this problem, LAMPE defines two (abstract) classes: DistributionModule and TransformModule. The former is any nn.Module whose forward method returns a PyTorch
Distribution. Similarly, the latter is any nn.Module whose forward method returns a PyTorch Transform. Then, what is a normalizing flow? It is simply a nn.Module that is constructed from a base DistributionModule and a list of TransformModule.

This design allows for very concise implementations of quite complex flows. Currently, LAMPE implements masked autoregressive flow (MAF), neural spline flow (NSF), neural autoregressive flow (NAF) and NAF based on unconstrained monotonic neural network (UMNN). All these flows support coupling (2 passes for inverse), fully autoregressive (as many passes as features) or anything in between (see Graphical Normalizing Flows). And all of that in about 800 lines of code, including whitespace and documentation. If you are interested, take a look at the transformations and flows.

Here is a small example with a neural spline flow (NSF).

>>> import lampe
>>> flow = lampe.nn.flows.NSF(7, context=16, transforms=3, hidden_features=[64] * 3, activation='ELU')
>>> flow
NSF(
  (transforms): ModuleList(
    (0): SoftclipTransform(bound=5.0)
    (1): MaskedAutoregressiveTransform(
      (base): MonotonicRQSTransform(bins=8)
      (order): [0, 1, 2, 3, 4, 5, 6]
      (params): MaskedMLP(
        (0): MaskedLinear(in_features=23, out_features=64, bias=True)
        (1): ELU(alpha=1.0)
        (2): MaskedLinear(in_features=64, out_features=64, bias=True)
        (3): ELU(alpha=1.0)
        (4): MaskedLinear(in_features=64, out_features=64, bias=True)
        (5): ELU(alpha=1.0)
        (6): MaskedLinear(in_features=64, out_features=161, bias=True)
      )
    )
    (2): MaskedAutoregressiveTransform(
      (base): MonotonicRQSTransform(bins=8)
      (order): [6, 5, 4, 3, 2, 1, 0]
      (params): MaskedMLP(
        (0): MaskedLinear(in_features=23, out_features=64, bias=True)
        (1): ELU(alpha=1.0)
        (2): MaskedLinear(in_features=64, out_features=64, bias=True)
        (3): ELU(alpha=1.0)
        (4): MaskedLinear(in_features=64, out_features=64, bias=True)
        (5): ELU(alpha=1.0)
        (6): MaskedLinear(in_features=64, out_features=161, bias=True)
      )
    )
    (3): MaskedAutoregressiveTransform(
      (base): MonotonicRQSTransform(bins=8)
      (order): [0, 1, 2, 3, 4, 5, 6]
      (params): MaskedMLP(
        (0): MaskedLinear(in_features=23, out_features=64, bias=True)
        (1): ELU(alpha=1.0)
        (2): MaskedLinear(in_features=64, out_features=64, bias=True)
        (3): ELU(alpha=1.0)
        (4): MaskedLinear(in_features=64, out_features=64, bias=True)
        (5): ELU(alpha=1.0)
        (6): MaskedLinear(in_features=64, out_features=161, bias=True)
      )
    )
    (4): Inverse(SoftclipTransform(bound=5.0))
  )
  (base): DiagNormal(loc: torch.Size([7]), scale: torch.Size([7]))
)

The flow is currently a nn.Module. To condition the flow with respect to a context y, we call it. This returns a distribution which can be evaluated (log_prob) or sampled (sample) just like any torch distribution.

>>> y = torch.randn(16)
>>> conditioned = flow(y)
>>> conditioned.sample()
tensor([ 1.1381,  0.3619, -1.9963,  0.2681, -0.1613,  0.1885, -0.4108])
>>> conditioned.sample((5, 6)).shape
torch.Size([5, 6, 7])
>>> x = torch.randn(7)
>>> conditioned.log_prob(x)
tensor(-8.6289, grad_fn=<AddBackward0>)
>>> x = torch.randn(5, 6, 7)
>>> conditioned.log_prob(x).shape
torch.Size([5, 6])

About time series data

Has the author tried to apply normalizing flows to the temporal prediction field? Or use normalizing flows to time series data?

Add stochastic NFs paper

Add stochastic normalizing flows paper. Adds stochasticity to your NF to help do better sampling.

• https://arxiv.org/abs/2002.06707

Video tutorial - Normalizing Flows - Motivations, The Big Idea, & Essential Foundations

https://www.youtube.com/watch?v=IuXU2dBOJyw

Hi @janosh,

I just uploaded a comprehensive tutorial on YouTube.

Since it is my video, I am not sending the pull request and request you to assess if it is good enough to be part of this repository.

Regards & thanks
Kapil

NF resource recommendation

Variational Inference using Normalizing Flows (VINF)

PDF: https://pierresegonne.github.io/VINF/
Medium: https://medium.com/swlh/normalizing-flows-are-not-magic-22752d0c924
GitHub: https://github.com/pierresegonne/VINF

New Pytorch package - Jammy Flows

I would like to add the package jammy_flows to the collection (https://github.com/thoglu/jammy_flows). I developed it for normalizing-flow inference in the context of astro-particle physics (https://arxiv.org/abs/2008.05825), but I think it might be useful in general.

It stands for Joint Autoregressive Manifold (MY) flows and models a (conditional) PDF on a tensor product of manifolds. The sub-PDFs are connected autoregressively similar to Inverse Autoregressive Flows (IAF - Kingma et al. 2016), but compared to IAF generalize via 1) allowing for arbitrary (non-affine) coupling layers (every flow in the package is amortizable) and 2) because we have general couplings we can link flows on different manifolds (e.g. Euclidean and a sphere).

It is mostly designed for low-dimensional applications (maybe a few 10s of dimensions - although simple flows ĺike the affine flow should reasonably work at much higher dimensionality) and should be simple to set up.

For example, a 5-d PDF defined on a 3-dimensional Euclidean manifold and an autoregressively linked 2-sphere conditional PDF is defined like this (together this forms a joint distribution on the tensor product space $\mathcal{R}^3 \times \mathcal{S}^2)$:

import jammy_flows

pdf=jammy_flows.pdf("e3+s2", "gg+n")

The first argument defines the autoregressive manifold structure, the second argument the exact flow layer used for each manifold.
Each flow is abbreviated with a letter (see below) - for example "g" stand for a Gaussinization flow layer and "n" for an autoregressive flow on the 2-sphere. The autoregressive connectivity and amortization (for conditional PDFs) is taken care of by the module, and in the configuration indicated by a "+" connecting the different options.

Without much tuning, you should get something that just works and has the properties of a flexible PDF... however high customization of flow parameters and connectivity is also possible if desired.

It implements a few state of the art flows that to my knowledge are not really found in other repositories yet (e.g. Gaussianization flows).

Currently implemented manifolds with respective flows (from the README):

Euclidean flows:

• Generic affine flow (Multivariate normal distribution) ("t")
• Gaussianization flow arXiv:2003.01941 ("g")
• Hybrid of nonlinear scalings and rotations ("Polynomial Stretch flow") ("p")

1-sphere Flows:

• Moebius transformations (described in arXiv:2002.02428) ("m")
• Circular rational-quadratic splines (described in arXiv:2002.02428) ("o")

2-sphere Flows:

• Autorregressive flow for N-Spheres (arXiv:2002.02428) ("n")
• Exponential map flow (arXiv:0906.0874/arXiv:2002.02428) ("v")
• Neural Manifold Ordinary Differential Equations arXiv:2006.10254 ("c")

Interval Flows:

• "Neural Spline Flows" (Rational-quadratic splines) arXiv:1906.04032 ("r")

Simplex Flows:

• Autoregressive simplex flow arXiv:2008.05456 ("w")

All of those can be combined in any way and the package automatically manages the connectivity.
More info can also be found in the docs.

Best,
Thorsten

Back fill date_added

Would be good to go back in git history and set the date_added field on some of the older items in this collection.

date_added should be mandatory for all newly added items.

Related #44.

I write on normalizing flows on my website and am wondering if it is fitted for this repo

You can check it at https://amaanv.com/articles under the headin 'Normalizing Flows'. I do it as a hobby and intend to write more. If you think it adds value to the repo I would love it if you can add it.

List stuff in reverse chronological order (newest to oldest)

Hi! This page is awesome (hence the name :) However it would be even more useful if you listed things (especially code) newest first, instead of oldest first, ie reverse chronological order.

Video addition to list Normalizing Flows: An Introduction and Review of Current Methods

Great overview!
Thanks for the effort

Is there a spot for the video tutorial created by one of the autohers of Normalizing Flows: An Introduction and Review of Current Methods and presented at ECCV2020?