americast / algo_bias

Detecting bias in ML models using heat maps

Algorithmic Bias

The aim of this project is to classify the income category people in a binary fashion as per their attributes. Next, we try to find out the factors which have contributed to the output of our model, at every layer, using a technique called Layerwise Relevance Propagation (LRP). Three variants of the implementation have been presented in this project. main.py is PyTorch based and tmain.py is TensorFlow based. LRP is supported by kmain.py and interprettensor/ which are in Keras (preferably through theano) and interprettonsor TF wrapper respectively. We also generate heatmaps for various layers against the input using the Keras variant.

Setup

pip install theano innvestigate

The package innvestigate installs a variant of keras which needs a version of tensorflow which is currently unsupported. Hence, we use theano for this project. We adjust the backend parameter to Theano instead of Tensorflow accordingly at ~/.keras/keras.json. For more information on how to do this, visit here.

Understanding the Code

The Keras variant of this code is used to generate heatmap as PyTorch and Tensorflow are not supported yet. So we will describe the kmain.py here until then. The first step is to import all the necessary libraries and functions. Next we define Batch Size, Epochs and Learning Rate to aid us in the training process. Then we use the Keras Sequential function to define model layers. Why we chose this particular model architecture? We will provide an ablation study later in this README listing accuracies of various models.

Next, we ask user for the choice to load previous saved model. Depending on which we load the model or train from beginning. We then ask user the choice to train or test. Depending on the selection, we load a csv file containing data to train or test. We then seperate attributes with target values, namely x_train and y_train. We then compile the model using Adam optimizer and Categorical Crossentropy as loss function. We use Keras' checkpoints to define early stopping (for training) if accuracy does not improve after certain epochs. Then depending on the user choice we train or test the model. If we train the model, then the best performing weights are saved to use them to load the model next time.

If the user chose to test, we print the model accuracy on the test data. Next, we do LRP (Layer-wise Relevance Propagation) which is a method for understanding deep neural networks by running a backward pass to observe how the individual layers of the program work. This technique brings brings explainability to Deep Neural Networks. More information on LRP is given here.

Source: heatmapping.org

Let us understand the code used to implement LRP using Innvestigate:

# we use lrp.z here out of various methods available like 
# Saliency, Deconvnet, GuidedBackprop, SmoothGrad, and IntergratedGradient
# then we pass our model here
analyzer = innvestigate.create_analyzer("lrp.z", model)
# then we pass model inputs, that is x_train
analysis = analyzer.analyze(x_train)

Then we perform the layer wise analysis by blocking off the layers and then performing analysis on them.

# Here we cutoff the layers and define a new model. This will remove the last Dense layer.
new_model_1 = Model(model.inputs, model.layers[-3].output)
# model.get_weights() return a numpy list of weights
new_model_1.set_weights(model.get_weights())
# summarize the model
new_model_1.summary()

Then we perform the analysis on this new model:

analyzer = innvestigate.create_analyzer("lrp.z", new_model_1)
analysis = analyzer.analyze(x_train)
# We save the results as numpy vectors
np.save("out_lrp", analysis)

We perform this method on every layers, meanwhile saving the numpy vectors.

Using this, we get layer by layer heatmaps in form of numpy vectors. In summary, we are cutting off layers in the network to obtain heatmap vectors for every layer and we are then merging them together to form a heatmap matrix/tensor. We use lrp_matrix.py to get complete numpy vectors then we plot them.

Choosing Appropriate Model

We perforemed experiments with various model architecture to choose the right one. We present a table below stating the model architecture and accuracy.

Dense layer : 'D', Batch Normalization: 'BN', Dropout: 'DR'

Model Architecture	Train acurracy	test accuracy
D512-SELU-D256-BN-SELU-D128-BN-SELU-D64-BN-SELU-D64-BN-SELU-D32-BN-SELU-D16-BN-SELU-D8-BN-SELU-D2-SIGMOID	79.64	42
D512-SELU-D512-BN-SELU-D128-BN-SELU-D64-BN-SELU-D64-BN-SELU-D32-BN-SELU-D16-BN-SELU-D8-BN-SELU-D2-SIGMOID	81.19	78.643
D512-SELU-D512-BN-SELU-D128-BN-SELU-D128-BN-SELU-D64-BN-SELU-D32-BN-SELU-D16-BN-SELU-D8-BN-SELU-D2-SIGMOID	80.37	59.787
D512-SELU-D512-BN-SELU-D512-BN-SELU-D128-BN-SELU-D64-BN-SELU-D32-BN-SELU-D16-BN-SELU-D8-BN-SELU-D2-SIGMOID	80.38	57.14
D256-RELU-D256-BN-RELU-D128-BN-RELU-D64-BN-RELU-D64-BN-RELU-D32-BN-RELU-D16-BN-RELU-D8-BN-RELU-D2-SIGMOID	80.20	77.25
D512-RELU-D512-BN-RELU-D256-BN-RELU-D64-BN-RELU-D64-BN-RELU-D32-BN-RELU-D16-BN-RELU-D8-BN-RELU-D2-SIGMOID	80.04	61.16
SAME PREVIOUS MODEL, EPOCHS=20000	92.477	78.84
D256-RELU-D256-BN-RELU-D128-BN-RELU-D128-BN-RELU-D64-BN-RELU-D64-BN-RELU-D32-BN-RELU-D16-BN-RELU-D8-BN-RELU-D2-SIGMOID	91.545	78.68
D512-RELU-D512-BN-RELU-DR(0.4)-D256-BN-RELU-DR(0.4)-D256-BN-RELU-DR(0.4)-D128-BN-RELU-DR(0.3)-D128-BN-RELU-DR(0.3)-D64-BN-RELU-DR(0.3)-D64-BN-RELU-DR(0.3)-D64-BN-RELU-DR(0.3)-D64-BN-RELU-DR(0.3)-D8-BN-RELU-DR(0.3)-D2-SIGMOID, EPOCHS=30000	88.23	81.78
D512-RELU-D256-BN-RELU-D256-BN-RELU-D256-BN-RELU-D128-BN-RELU-D64-BN-RELU-D64-BN-RELU-D16-BN-RELU-D8-BN-RELU-D2-SIGMOID	91.361	77.86

Generating heatmaps

Next, we generate heatmaps using

python kmain.py
python lrp_matrix.py
python plot.py

Please make sure you run kmain.py in inference mode by answering the questions prompted accordingly.

Sample heatmap

algo_bias's Issues

Drop unnecessary data while preprocessing

Dropping of unnecessary Scale_ID takes place while training. This can be done while preprocessing to speed up training.

Improve README

Improve the README describing the flow in kmain.py, and how LRP is being used to obtain the contributions from the inputs at every layer.

To-Dos :thinking:

• Distribute train and test data 🙄
• Add tanh activation 🎃
• Change layer sizes 🤕

Modularise the keras variant

The heatmaps are generated using keras variant kmain.py. It would be nice to have:

• More modularity in the code (by adding more functions)
• Docstrings for every module.

Create a gif for the heatmap

The heatmap generated is static and is based on the final trained model. It would be great to have different heatmaps based on the current iteration of training. A gif as a collection of all such heatmaps on the README would help in better analysis.

Make the code reproducible

In order to improve code reproducibility, it would be great to have a fixed random seed for all the experiments. Experimentations may also be done using different seeds, however, they need to be updated in the README accordingly.

Improve accuracy of the trained model

It would be nice to experiment with other loss functions and alternate network architectures. Also, results in the form of an ablation study in the README would be great.