microsoft / horseshoe-bnn

A Bayesian Neural Network with a horseshoe prior for improved interpretability

Interpretable Outcome Prediction with Sparse Bayesian Neural Networks in Intensive Care

This repository contains all code for the paper Interpretable Outcome Prediction with Sparse Bayesian Neural Networks in Intensive Care which can be found here.

Abstract: Clinical decision making is challenging because of pathological complexity, as well as large amounts of heterogeneous data generated by various medical instruments. In recent years, machine learning tools have been developed to aid clinical decision making. Especially models for outcome prediction in intensive care units (ICUs) could help, for example, with decisions about resource allocation. However, flexible tools such as deep neural networks are rarely deployed in healthcare systems due to a lack of interpretability. In this work, we propose a novel interpretable Bayesian neural network (BNN) architecture, which offers both the flexibility of artificial neural networks and interpretability in terms of feature selection. In particular, we employ a sparsity inducing prior distribution in a tied manner to learn which features are important for performing outcome prediction. In addition, our model is fully probabilistic. Thus, the model provides probabilistic outcome prediction results, as well as information about the importance of different input features (i.e. clinical measurements). We evaluate our approach on the task of mortality prediction using two cohorts from real-world intensive care units. Collaborating with healthcare experts, we found that our approach can provide novel insights into the importance of different clinical measurements, in addition to the predicted outcome results. This suggests that our model can support medical experts in their decision making.

Table of Contents

1. Installation
2. Citation
3. Datasets
4. Getting started
5. Structure of repository
6. License
7. Contributing

Installation

The code runs in Python 3.7 and Pytorch 1.1. To install the package please first clone the repo. After that, navigate to the project and run the following command in your command line. This will install the horseshoe_bnn package.

pip install -e .

Other requirements can be installed using the requirements file located in the repo:

pip install -r requirements.txt

Citation

If you use this code in your research, please cite the following publication:

Bibtex

@article{popkes2019, title={Interpretable Outcome Prediction with Sparse Bayesian Neural Networks in Intensive Care}, author={Popkes, Anna-Lena and Overweg, Hiske and Ercole, Ari and Li, Yingzhen and Hern{'a}ndez-Lobato, Jos{'e} Miguel and Zaykov, Yordan and Zhang, Cheng}, journal={arXiv preprint arXiv:1905.02599}, year={2019} }

MLA

Popkes, Anna-Lena, et al. "Interpretable Outcome Prediction with Sparse Bayesian Neural Networks in Intensive Care." arXiv preprint arXiv:1905.02599 (2019).

APA

Popkes, A. L., Overweg, H., Ercole, A., Li, Y., Hernández-Lobato, J. M., Zaykov, Y., & Zhang, C. (2019). Interpretable Outcome Prediction with Sparse Bayesian Neural Networks in Intensive Care. arXiv preprint arXiv:1905.02599.

Datasets

1. MIMIC-III

• Instructions on how to download the dataset can be found here
• Details on the dataset, including preprocessing and features can be found in our paper

2. Center TBI dataset

• This dataset is not publically available
• Details on the dataset, including preprocessing and features can be found in our paper

3. Boston Housing Dataset

• To demonstrate the functioning of our networks and evaluator we use a publically available dataset, available within sckikit-learn
• The example script for running the evaluator uses this dataset

Getting started

A script which trains and evaluates all models on the Boston Housing dataset can be found here. The script contains an example of how to train the models using the evaluator. To run the example script please follow the installation instructions. After that, navigate to the folder horseshoe_bnn/evaluation/ and run the following command. This will run all models on the Boston Housing dataset and print results to the console.

python evaluate_all_models.py

The code for the individual models is located here.

Repository Structure

├── LICENSE
├── README.md          <- The top-level README for developers using this project.
│
├── requirements.txt   <- The requirements file for reproducing the analysis environment
│
├── test               <- All pytest test code
│
├── mimic_preprocessing <- Scripts to preprocess the MIMIC-III dataset
│
├── horseshoe_bnn       <- All source code
│    │
│    ├── __init__.py    <- Makes horseshoe_bnn a Python module
│    │
│    ├── data_handling        <- Scripts to handle data
│    │   │                
│    │   ├── dataset.py                <- Custom Dataset class
│    │   └── dataset_to_dataloaders.py <- Function to convert Dataset instance to Pytorch dataloader
│    │
│    │
│    ├── evaluation     <- Scripts to evaluate models
│    │   │                 
│    │   ├── evaluator.py           <- Evaluator
│    │   └── evaluate_all_models.py <- Example script how to run evaluation
│    │
│    ├── metrics.py              <- Metric classes
│    ├── models.py               <- All model classes
│    ├── distributions.py        <- Distribution classes
│    ├── network_layers.py       <- BNN network layer classes
│    ├── aggregation_result.py   <- Aggregation result class
│    └── parameters.py           <- Parameter classes
│
└── 

License

This repository is licensed under the Microsoft Research license.

Contributing

This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.microsoft.com.

When you submit a pull request, a CLA-bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., label, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact [email protected] with any additional questions or comments.

horseshoe-bnn's Issues

Variational distribution over weights of horseshoe layer

Hi,

I wonder if the following code is different from the description in the paper.

horseshoe-bnn/horseshoe_bnn/network_layers.py

Lines 232 to 236 in 6db8475

	beta = self.beta.sample(n_samples)
	log_tau = torch.unsqueeze(self.log_tau.sample(n_samples), 1)
	log_v = torch.unsqueeze(self.log_v.sample(n_samples), 1)
	weight = beta * log_tau * log_v

One of the reasons to use log-normal distribution over $\tau$ and $v$ is to make sure they are nonnegative. I think the correct version would be

weight = beta * log_tau.exp() * log_v.exp()

Thank you for the nice paper, source code.

How do you plot your weight?

Hi,

I was trying to use your model on my own dataset to do the feature selection. Since I don't have access to the dataset you described in the paper, I used the Boston dataset in your repo. But I found that the feature importance scores calculated by the norm of weights are not consistent across the 5-folds, as well as not consistent across the models (HorseshoeBNN and LinearHorseshoe).

For example, comparing the weights of the first layer between HorseshoeBNN and LinearHorseshoe for the first fold of train-test-split (which means they are trained using the same dataset):

Another example compares the weights of the first layer of HorseshoeBNN between the first and second fold of train-test-split:

However, Figure 8 in the paper shows that the weight distributions are very consistent as least between HorseshoeBNN and LinearHorseshoe. I was wondering if I'm doing anything wrong with plotting the weight distributions. Basically I just sample the beta, tau, and v from the first Horseshoe layer and multiply them together according to weights = beta * tau * v, and calculate the mean and std for the weights. My code for plotting the norm of weights as the feature importance scores is attached:

def plot_weights_dist(means: np.ndarray, variances: np.ndarray, feature_names: list) -> None:
    y_pos = np.arange(len(feature_names))
    fig, ax = plt.subplots()
    ax.barh(y_pos, np.abs(means), xerr=variances)
    ax.set_xlabel('|Weight|')
    ax.set_ylabel('Feature Names')
    plt.yticks(y_pos, feature_names)
    plt.title("Probability Distribution for Each Weight")
    plt.show()

feature_names = ["x1", "x2", "x3", "x4", "x5", "x6", "x7", "x8", "x9", "x10", "x11", "x12", "x13"]
n_samples = config['n_samples_testing']
beta = model.l1.beta.sample(n_samples) # for HorseshoeBNN use model.l1, for LinearHorseshoe use model.layer, the same to the next two line
log_tau = torch.unsqueeze(model.l1.log_tau.sample(n_samples), 1)
log_v = torch.unsqueeze(model.l1.log_v.sample(n_samples), 1)
weight = beta * log_tau * log_v
weight = weight.reshape(-1, weight.shape[-1])
plot_weights_dist(torch.mean(torch.abs(weight), dim=0).cpu().detach().numpy(),
                  torch.std(torch.abs(weight), dim=0).cpu().detach().numpy(),
                  feature_names)