yookoon / density_uncertainty_layers

Density Uncertainty Layers

This repository contains the PyTorch implementation for the paper "Density Uncertainty Layers for Reliable Uncertainty Estimation" published in AISTATS 2024.

Requirements

The code is implemented using PyTorch 1.12.1.

To install the required packages (other than PyTorch)

pip install -r requirements.txt

Running Experiments

To train Rank1 Density Uncertainty Layers WRN28 on CIFAR-10/100,

python run_cifar.py --dataset={cifar10/cifar100} --model=rank1_density_wrn28

To train other models, simply replace the model argument with one of

density_wrn28, bayesian_wrn28, mcdropout_wrn28, vdropout_wrn28, rank1_wrn28

For UCI benchmarks,

python run_uci.py --model={density_mlp/bayesian_mlp/mcdropout_mlp/vdropout_mlp/rank1_mlp}

Citation

@inproceedings{park2024,
  title={Density Uncertainty Layers for Reliable Uncertainty Estimation},
  author={Park, Yookoon and Blei, David},
  booktitle={AISTATS},
  year={2024}
}

density_uncertainty_layers's Issues

Computation of Predictive Uncertainty in Regression Setting

@yookoon Thank you for the interesting work and the code repository. We are trying to support your proposed method in our UQ-Library for Deep Learning called Lightning-UQ-Box and have a couple of questions regarding the regression framework.

To take the Regression case based on the UCI datasets for example, the MLP module has a logvar parameter that is a single value for the entire model. During the training and test phase the loss is computed with this homoscedastic logvar parameter here. Additionally, for sampling methods like BNNs or your proposed Density framework, N samples are taken in this loop, however, you only compute the average over the model mean predictions, and the sampling of the weights has no effect on the predictive uncertainty you are computing, since it is just a single learned parameter for the entire model. This seems counterintuitive as the point of BNNs is to model epistemic uncertainty which should influence the overall predictive uncertainty that I get from the model.

In Figure 1 of your paper, you show results on a Toy Regression dataset with input dependent uncertainty, however, the repository does not contain the code to generate this figure as far as I can tell. In your paper you state that based on equations 7-9 "Consequently, predictive uncertainty will be high for test inputs that are improbable in the training density and low for those that are more probable, providing intuitive and reliable predictive uncertainty." However, I fail to see how that can be the case, when you are using a single logvar parameter as your predictive uncertainty. I was therefore wondering whether you could help me out in understanding the utilized notion of predictive uncertainty in the regression case. Thanks in advance!