akhilpandey95 / uncertainty

Work as part of ANL summer 2020 research on uncertainity quanitification methods in graph neural networks

About

The repository consists of information assosicated with the research on uncertainity quantification for graph neural network architectures. This work was part of an internship at ANL in the Mathematics and Computer Sciences (MCS) division. The repository has 5 directories:

• data/ - The directory that consists of the datasets in the form of pickled files
• literature/ - Summaries of all the scholarly articles that are read through the course of the project.
• src/ - Code, documentation, and essential information about the models built for the project.
• scripts/ - Essential scripts necessary for setting up the project.
• tests/ - Deterministic unit tests for better replication of the results.
• notebooks/ - Jupyter notebook(s) with sample code showcasing how to utilize the resources of the project.

Installation

Install the appropriate python packages

pip install -r requirements.txt

Data

We have used two datasets. The idea was to perform uncertainty quantification on Fully Connected Networks, and Graph Neural Networks. So, for the Fully Connected Networks we generated a sin dataset of sample size 1201 with gaussian noise given by, y = 10 sin(x) + ε. The dataset looks like this:

For the Graph Neural Networks, we have used the Quantum Chemistry dataset, QM7.

Both of the datasets have single output and would be mapped as regression problems.

Experiments

We built simple deep neural networks for predicting the response variable. We then estimated the uncertainties in the models predicting the response values. We tried different approaches and as a result some of the methods that we used supported quantification of both aleatoric, and epistemic uncertainties, while the remaining weren't equipped for the same. The approaches we have tried could potentially be categorized into ones that are capable of estimating both aleatoric, and epistemic uncertainties, and others that can only estimate epistemic uncertainties.

Methods that can approximate both aleatoric and epistemic uncertainty :

• Bayesian Neural Network (BNN) : Bayesian Learning for Neural Networks by R.M. Neal
• SQR+ONC : Single-Model Uncertainties for Deep Learning by N.Tagasovska, and D. Lopez-Paz

---

Methods that can approximate only epistemic uncertainty :

• DeepEnsembles : Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles by B. Lakshminarayanan et.al
• MC Dropout : Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning by Y. Gal, and Z. Ghahramani
• Ensemble MC Dropout : Understanding Measures of Uncertainty for Adversarial Example Detection by L. Smith, and Y. Gal

For the models predicting the y values of the sin dataset, we have used a simple one hidden layer fully connected neural network. The following are the architectural details:

Model	Optimum hyper parameters
Epochs	200, 1500 for BNN
Batch size	128
Loss Function	MSE, Gaussian Loss for DeepEnsemble, Negative log likelihood for BNN, Pinball Loss for SQR+ONC
Hidden Layers	1 layer with 256 neurons
Optimization function	Adam with 0.001 learning rate
Activation function(s)	ReLU for the hidden layer, Linear for the o/p layer

Results

Linear regression on the sin dataset:

Model	MSE	MAE	RMSE	R2	Training time	Inference Time
Fully Connected Network (FCN)	1.0053	0.8078	1.0026	0.9813	33.4 s	246 ms
FCN with MC Dropout	1.4041	0.9555	1.1849	0.9740	15.7 s	54.7 s
FCN with Ensemble MC Dropout	1.4430	0.9697	1.2012	0.9733	5min 56s	32min 19s
FCN with DeepEnsembles	0.9531	0.7767	0.9763	0.9820	10min 25s	1min 50s
FCN with SQR, and ONC	1.0929	0.8318	1.0454	0.9793	43s	10.2s
Bayesian Neural Network	3.3879	1.4489	1.9418	0.953	8min 39s	44s

Linear regression on the QM7 dataset :

On the test set,

Model	MSE	MAE	RMSE	R2	Training Time	Inference Time
Graph Convolutional Network (GCN)	15498.9595	75.9521	124.4948	0.6830	2min 30s	444 ms
GCN with MC Dropout	23116.5562	119.5583	152.0412	0.5272	1min 51s	8.46 s
GCN with Ensemble MC Dropout	23382.3769	121.7387	152.9129	0.5218	1h 45s	6min 46s
GCN with DeepEnsembles	15726.9933	76.9819	125.3840	0.6783	1h 3min 15s	2min 59s
GCN with SQR, and ONC	21182.1737	99.7401	145.5409	0.5668	2min 30s	579 ms
Bayesian Graph Neural Network	-	-	-	-

On the validation set,

Model	MSE	MAE	RMSE	R2	Inference Time
Graph Convolutional Network (GCN)	10658.5007	68.3195	103.2400	0.7811	153 ms
GCN with MC Dropout	17058.3652	106.2512	130.6076	0.6497	7.99 s
GCN with Ensemble MC Dropout	18296.0994	112.2832	135.2630	0.6243s	3min 36s
GCN with DeepEnsembles	10529.2807	67.4484	102.5716	0.7838	4.01 s
GCN with SQR, and ONC	17238.2398	94.7970	131.2944	0.6460	427 ms
Bayesian Graph Neural Network	-	-	-	-

Author

Akhil Pandey

Mentor

Prasanna Balaprakash

uncertainty's Issues

Update results section in the README

• add results for the GCN experiments

Add the script for preparing the qm7 dataset

• add about the scripts/ directory in README.md
• name the script prepare_qm7_data.py

Add summaries of all the references in the literature directory

• create a markdown file for every summary and index the summaries in the same order as the references listed.

Build FCN model and run regular NN, and dropout on Uno

• build a simple model and run experiment on Uno.
• build a simple model with MC dropout and run experiment on Uno.
• build a simple model with Ensemble MC dropout and run experiment on Uno.

Add smilesLSTM model architecture to the repo

For architecture refer :

• https://pubs.rsc.org/en/content/articlepdf/2018/sc/c8sc00148k
• http://www.rsc.org/suppdata/c8/sc/c8sc00148k/c8sc00148k1.pdf

add literature for uncertainity quantification methods in general

Add all the UQ metrics as helper methods to evaluate.py

Add the network diagram that summarized the literature on UQ

• add the diagram to literature/media/

add literature for uncertainity quantification methods in graph neural networks

Train the Bayesian SmilesLSTM model and collect results for UQ

Implement UQ on the SmilesLSTM architecture using bayesian neural network

Implement Bayesian Graph Convolutional Neural Network in TF2

paper : https://arxiv.org/pdf/1811.11103.pdf

Add a readme and gitignore

• highlight the structure of the repository
• include bare minimum items in gitignore

Upload markdown notebooks for all trained examples using the toy data

Init src/ with base structure of the repo

• data.py
• util.py
• models.py
• evaluation.py

Add results for the experiments on the Quantum Chemistry datasets

• QM8
• QM9

Add data loader methds for loading Uno

• Write a method that can use CombinedDataLoader function to take file inputs instead of command line arguments
• Write method that can use CombinedDataGenerator function to create batch inputs from files.
• Customize DataFeeder to work on file inputs directly

Add the pickled files for the toy datasets

• update the README.md about the data/ directory
• data/sin for the toy sin data set with noise
• data/qm7 for the toy graph data set

Add a tf2.0 implementation of Graph convolutional network

Implement UQ on SmilesLSTM using deep ensembles

Build FCN model to run Deepensembles on Uno

• Extend the GaussianLayer to the simple FCN Uno model from src/layers.py and do deepensembles

Add summary for literature on bayesian recurrent networks

• Paper : https://arxiv.org/pdf/1704.02798.pdf

Write a class object for the gaussian layer used in Deep ensembles

Perform UQ for graph neural networks using toy datasets

Update README with details about the project

• add the icon
• add the salient points from the experiments
• add the plots for all the methods

Add y true vs y predicted plots for all the single class regression experiments

• ESOL
• FreeSolv
• Lipophilicity
• QM7

Add summart for litearture on deep ensembles for UQ

• Paper : https://arxiv.org/pdf/1612.01474.pdf

Add results for the experiments on Physical chemsitry datasets

• ESOL
• FreeSolv
• Lipophilicity

Add a notebook comprising of all experiments made on the toy dataset

y = 10 sin(x) + ε

Write summary for the Bayesian Graph convolutional Network paper

link : https://arxiv.org/pdf/1811.11103.pdf

Add data processing methods for the drug discovery datasets

• sars_cov - http://www.bioinf.jku.at/people/klambauer/sars_cov_dataset.zip
• LSC - http://bioinf.at/research/lsc/
• Tox21 - http://bioinf.jku.at/research/DeepTox/tox21.html

Train the DeepEnsemble based SmileLSTM model and collect results for UQ

Add custom implementations of Graph layers used in deepchem

add a custom TF2 implementation of :

• GraphConv layer
• GraphPool layer
• GraphGather layer

Source : https://github.com/deepchem/deepchem/blob/tf/deepchem/models/layers.py