slundberg / shap

A game theoretic approach to explain the output of any machine learning model.

---

SHAP (SHapley Additive exPlanations) is a game theoretic approach to explain the output of any machine learning model. It connects optimal credit allocation with local explanations using the classic Shapley values from game theory and their related extensions (see papers for details and citations).

Install

SHAP can be installed from either PyPI or conda-forge:

pip install shap
or
conda install -c conda-forge shap

Tree ensemble example (XGBoost/LightGBM/CatBoost/scikit-learn/pyspark models)

While SHAP can explain the output of any machine learning model, we have developed a high-speed exact algorithm for tree ensemble methods (see our Nature MI paper). Fast C++ implementations are supported for XGBoost, LightGBM, CatBoost, scikit-learn and pyspark tree models:

import xgboost
import shap

# train an XGBoost model
X, y = shap.datasets.california()
model = xgboost.XGBRegressor().fit(X, y)

# explain the model's predictions using SHAP
# (same syntax works for LightGBM, CatBoost, scikit-learn, transformers, Spark, etc.)
explainer = shap.Explainer(model)
shap_values = explainer(X)

# visualize the first prediction's explanation
shap.plots.waterfall(shap_values[0])

The above explanation shows features each contributing to push the model output from the base value (the average model output over the training dataset we passed) to the model output. Features pushing the prediction higher are shown in red, those pushing the prediction lower are in blue. Another way to visualize the same explanation is to use a force plot (these are introduced in our Nature BME paper):

# visualize the first prediction's explanation with a force plot
shap.plots.force(shap_values[0])

If we take many force plot explanations such as the one shown above, rotate them 90 degrees, and then stack them horizontally, we can see explanations for an entire dataset (in the notebook this plot is interactive):

# visualize all the training set predictions
shap.plots.force(shap_values[:500])

To understand how a single feature effects the output of the model we can plot the SHAP value of that feature vs. the value of the feature for all the examples in a dataset. Since SHAP values represent a feature's responsibility for a change in the model output, the plot below represents the change in predicted house price as the latitude changes. Vertical dispersion at a single value of latitude represents interaction effects with other features. To help reveal these interactions we can color by another feature. If we pass the whole explanation tensor to the color argument the scatter plot will pick the best feature to color by. In this case it picks longitude.

# create a dependence scatter plot to show the effect of a single feature across the whole dataset
shap.plots.scatter(shap_values[:, "Latitude"], color=shap_values)

To get an overview of which features are most important for a model we can plot the SHAP values of every feature for every sample. The plot below sorts features by the sum of SHAP value magnitudes over all samples, and uses SHAP values to show the distribution of the impacts each feature has on the model output. The color represents the feature value (red high, blue low). This reveals for example that higher median incomes improves the predicted home price.

# summarize the effects of all the features
shap.plots.beeswarm(shap_values)

We can also just take the mean absolute value of the SHAP values for each feature to get a standard bar plot (produces stacked bars for multi-class outputs):

shap.plots.bar(shap_values)

Natural language example (transformers)

SHAP has specific support for natural language models like those in the Hugging Face transformers library. By adding coalitional rules to traditional Shapley values we can form games that explain large modern NLP model using very few function evaluations. Using this functionality is as simple as passing a supported transformers pipeline to SHAP:

import transformers
import shap

# load a transformers pipeline model
model = transformers.pipeline('sentiment-analysis', return_all_scores=True)

# explain the model on two sample inputs
explainer = shap.Explainer(model)
shap_values = explainer(["What a great movie! ...if you have no taste."])

# visualize the first prediction's explanation for the POSITIVE output class
shap.plots.text(shap_values[0, :, "POSITIVE"])

Deep learning example with DeepExplainer (TensorFlow/Keras models)

Deep SHAP is a high-speed approximation algorithm for SHAP values in deep learning models that builds on a connection with DeepLIFT described in the SHAP NIPS paper. The implementation here differs from the original DeepLIFT by using a distribution of background samples instead of a single reference value, and using Shapley equations to linearize components such as max, softmax, products, divisions, etc. Note that some of these enhancements have also been since integrated into DeepLIFT. TensorFlow models and Keras models using the TensorFlow backend are supported (there is also preliminary support for PyTorch):

# ...include code from https://github.com/keras-team/keras/blob/master/examples/demo_mnist_convnet.py

import shap
import numpy as np

# select a set of background examples to take an expectation over
background = x_train[np.random.choice(x_train.shape[0], 100, replace=False)]

# explain predictions of the model on four images
e = shap.DeepExplainer(model, background)
# ...or pass tensors directly
# e = shap.DeepExplainer((model.layers[0].input, model.layers[-1].output), background)
shap_values = e.shap_values(x_test[1:5])

# plot the feature attributions
shap.image_plot(shap_values, -x_test[1:5])

The plot above explains ten outputs (digits 0-9) for four different images. Red pixels increase the model's output while blue pixels decrease the output. The input images are shown on the left, and as nearly transparent grayscale backings behind each of the explanations. The sum of the SHAP values equals the difference between the expected model output (averaged over the background dataset) and the current model output. Note that for the 'zero' image the blank middle is important, while for the 'four' image the lack of a connection on top makes it a four instead of a nine.

Deep learning example with GradientExplainer (TensorFlow/Keras/PyTorch models)

Expected gradients combines ideas from Integrated Gradients, SHAP, and SmoothGrad into a single expected value equation. This allows an entire dataset to be used as the background distribution (as opposed to a single reference value) and allows local smoothing. If we approximate the model with a linear function between each background data sample and the current input to be explained, and we assume the input features are independent then expected gradients will compute approximate SHAP values. In the example below we have explained how the 7th intermediate layer of the VGG16 ImageNet model impacts the output probabilities.

from keras.applications.vgg16 import VGG16
from keras.applications.vgg16 import preprocess_input
import keras.backend as K
import numpy as np
import json
import shap

# load pre-trained model and choose two images to explain
model = VGG16(weights='imagenet', include_top=True)
X,y = shap.datasets.imagenet50()
to_explain = X[[39,41]]

# load the ImageNet class names
url = "https://s3.amazonaws.com/deep-learning-models/image-models/imagenet_class_index.json"
fname = shap.datasets.cache(url)
with open(fname) as f:
    class_names = json.load(f)

# explain how the input to the 7th layer of the model explains the top two classes
def map2layer(x, layer):
    feed_dict = dict(zip([model.layers[0].input], [preprocess_input(x.copy())]))
    return K.get_session().run(model.layers[layer].input, feed_dict)
e = shap.GradientExplainer(
    (model.layers[7].input, model.layers[-1].output),
    map2layer(X, 7),
    local_smoothing=0 # std dev of smoothing noise
)
shap_values,indexes = e.shap_values(map2layer(to_explain, 7), ranked_outputs=2)

# get the names for the classes
index_names = np.vectorize(lambda x: class_names[str(x)][1])(indexes)

# plot the explanations
shap.image_plot(shap_values, to_explain, index_names)

Predictions for two input images are explained in the plot above. Red pixels represent positive SHAP values that increase the probability of the class, while blue pixels represent negative SHAP values the reduce the probability of the class. By using ranked_outputs=2 we explain only the two most likely classes for each input (this spares us from explaining all 1,000 classes).

Model agnostic example with KernelExplainer (explains any function)

Kernel SHAP uses a specially-weighted local linear regression to estimate SHAP values for any model. Below is a simple example for explaining a multi-class SVM on the classic iris dataset.

import sklearn
import shap
from sklearn.model_selection import train_test_split

# print the JS visualization code to the notebook
shap.initjs()

# train a SVM classifier
X_train,X_test,Y_train,Y_test = train_test_split(*shap.datasets.iris(), test_size=0.2, random_state=0)
svm = sklearn.svm.SVC(kernel='rbf', probability=True)
svm.fit(X_train, Y_train)

# use Kernel SHAP to explain test set predictions
explainer = shap.KernelExplainer(svm.predict_proba, X_train, link="logit")
shap_values = explainer.shap_values(X_test, nsamples=100)

# plot the SHAP values for the Setosa output of the first instance
shap.force_plot(explainer.expected_value[0], shap_values[0][0,:], X_test.iloc[0,:], link="logit")

The above explanation shows four features each contributing to push the model output from the base value (the average model output over the training dataset we passed) towards zero. If there were any features pushing the class label higher they would be shown in red.

If we take many explanations such as the one shown above, rotate them 90 degrees, and then stack them horizontally, we can see explanations for an entire dataset. This is exactly what we do below for all the examples in the iris test set:

# plot the SHAP values for the Setosa output of all instances
shap.force_plot(explainer.expected_value[0], shap_values[0], X_test, link="logit")

SHAP Interaction Values

SHAP interaction values are a generalization of SHAP values to higher order interactions. Fast exact computation of pairwise interactions are implemented for tree models with shap.TreeExplainer(model).shap_interaction_values(X). This returns a matrix for every prediction, where the main effects are on the diagonal and the interaction effects are off-diagonal. These values often reveal interesting hidden relationships, such as how the increased risk of death peaks for men at age 60 (see the NHANES notebook for details):

Sample notebooks

The notebooks below demonstrate different use cases for SHAP. Look inside the notebooks directory of the repository if you want to try playing with the original notebooks yourself.

TreeExplainer

An implementation of Tree SHAP, a fast and exact algorithm to compute SHAP values for trees and ensembles of trees.

• NHANES survival model with XGBoost and SHAP interaction values - Using mortality data from 20 years of followup this notebook demonstrates how to use XGBoost and shap to uncover complex risk factor relationships.

• Census income classification with LightGBM - Using the standard adult census income dataset, this notebook trains a gradient boosting tree model with LightGBM and then explains predictions using shap.

• League of Legends Win Prediction with XGBoost - Using a Kaggle dataset of 180,000 ranked matches from League of Legends we train and explain a gradient boosting tree model with XGBoost to predict if a player will win their match.

DeepExplainer

An implementation of Deep SHAP, a faster (but only approximate) algorithm to compute SHAP values for deep learning models that is based on connections between SHAP and the DeepLIFT algorithm.

• MNIST Digit classification with Keras - Using the MNIST handwriting recognition dataset, this notebook trains a neural network with Keras and then explains predictions using shap.

• Keras LSTM for IMDB Sentiment Classification - This notebook trains an LSTM with Keras on the IMDB text sentiment analysis dataset and then explains predictions using shap.

GradientExplainer

An implementation of expected gradients to approximate SHAP values for deep learning models. It is based on connections between SHAP and the Integrated Gradients algorithm. GradientExplainer is slower than DeepExplainer and makes different approximation assumptions.

• Explain an Intermediate Layer of VGG16 on ImageNet - This notebook demonstrates how to explain the output of a pre-trained VGG16 ImageNet model using an internal convolutional layer.

LinearExplainer

For a linear model with independent features we can analytically compute the exact SHAP values. We can also account for feature correlation if we are willing to estimate the feature covariance matrix. LinearExplainer supports both of these options.

• Sentiment Analysis with Logistic Regression - This notebook demonstrates how to explain a linear logistic regression sentiment analysis model.

KernelExplainer

An implementation of Kernel SHAP, a model agnostic method to estimate SHAP values for any model. Because it makes no assumptions about the model type, KernelExplainer is slower than the other model type specific algorithms.

• Census income classification with scikit-learn - Using the standard adult census income dataset, this notebook trains a k-nearest neighbors classifier using scikit-learn and then explains predictions using shap.

• ImageNet VGG16 Model with Keras - Explain the classic VGG16 convolutional neural network's predictions for an image. This works by applying the model agnostic Kernel SHAP method to a super-pixel segmented image.

• Iris classification - A basic demonstration using the popular iris species dataset. It explains predictions from six different models in scikit-learn using shap.

Documentation notebooks

These notebooks comprehensively demonstrate how to use specific functions and objects.

• shap.decision_plot and shap.multioutput_decision_plot

• shap.dependence_plot

Methods Unified by SHAP

1. LIME: Ribeiro, Marco Tulio, Sameer Singh, and Carlos Guestrin. "Why should i trust you?: Explaining the predictions of any classifier." Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, 2016.

2. Shapley sampling values: Strumbelj, Erik, and Igor Kononenko. "Explaining prediction models and individual predictions with feature contributions." Knowledge and information systems 41.3 (2014): 647-665.

3. DeepLIFT: Shrikumar, Avanti, Peyton Greenside, and Anshul Kundaje. "Learning important features through propagating activation differences." arXiv preprint arXiv:1704.02685 (2017).

4. QII: Datta, Anupam, Shayak Sen, and Yair Zick. "Algorithmic transparency via quantitative input influence: Theory and experiments with learning systems." Security and Privacy (SP), 2016 IEEE Symposium on. IEEE, 2016.

5. Layer-wise relevance propagation: Bach, Sebastian, et al. "On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation." PloS one 10.7 (2015): e0130140.

6. Shapley regression values: Lipovetsky, Stan, and Michael Conklin. "Analysis of regression in game theory approach." Applied Stochastic Models in Business and Industry 17.4 (2001): 319-330.

7. Tree interpreter: Saabas, Ando. Interpreting random forests. http://blog.datadive.net/interpreting-random-forests/

Citations

The algorithms and visualizations used in this package came primarily out of research in Su-In Lee's lab at the University of Washington, and Microsoft Research. If you use SHAP in your research we would appreciate a citation to the appropriate paper(s):

• For general use of SHAP you can read/cite our NeurIPS paper (bibtex).
• For TreeExplainer you can read/cite our Nature Machine Intelligence paper (bibtex; free access).
• For GPUTreeExplainer you can read/cite this article.
• For force_plot visualizations and medical applications you can read/cite our Nature Biomedical Engineering paper (bibtex; free access).

shap's Issues

SHAP vs LIME

how would you explain how your algorithm (in xgboost) is different than LIME?

Possible to Generate Summary Plots and Dependence Plots using SKLearn Models?

Hi,

Thanks for this wonderful resource! It has enhanced my work greatly.

I was wondering if it is possible to generate Summary and Dependence plots using SKLearn generated models. I have been able to duplicate your work using my own data trained on Sklearn's forest models. But I have not been able to create the plots in question. I also notice you don't include them in your sklearn examples.

Any help or hint is appreciated!

Thanks,

Keita

Summary plot for Keras NN

Hello,

I'm trying to visualize a summary plot for a NN model implemented using keras but I'm getting this error:

AttributeError Traceback (most recent call last)
in ()
----> 1 shap.summary_plot(shap_values2, test_data.iloc[:50,:])

~/anaconda3/lib/python3.6/site-packages/shap/plots.py in summary_plot(shap_values, features, feature_names, max_display, plot_type, color, axis_color, title, alpha, show, sort, color_bar, auto_size_plot)
262 """
263
--> 264 assert len(shap_values.shape) != 1, "Summary plots need a matrix of shap_values, not a vector."
265
266 # convert from a DataFrame or other types

AttributeError: 'list' object has no attribute 'shape'

is this visualization only available for XGBoost model or is it model agnostic?

shap.visualize() output charts are truncated / poorly formatted

Environment info

Firefox 52.5.2 on Debian 9.3

requirements.txt

Steps to reproduce

print(shap_values)
print(feature_names)
print(np.array(X))
shap.visualize(shap_values[0,:], feature_names=feature_names, data=np.array(X)[0,:])

Output:

[[-0.09046029 -0.05412534 -0.00113313]
 [-0.08825505  0.04928131 -0.00113313]
 [ 0.11093738 -0.05434655 -0.00113313]
 [ 0.10730842  0.04322589 -0.00113313]
 [-0.09046029 -0.05412534 -0.00113313]
 [-0.08825505  0.04928131 -0.00113313]
 [ 0.11093738 -0.05434655 -0.00113313]
 [ 0.10730842  0.04322589 -0.00113313]]
['TestFloatFeature', 'TestBinaryFeature']
[[0.2 2. ]
 [0.2 1. ]
 [0.4 2. ]
 [0.4 1. ]
 [0.2 2. ]
 [0.2 1. ]
 [0.4 2. ]
 [0.4 1. ]
 [0.2 2. ]
 [0.2 1. ]
 [0.4 2. ]
 [0.4 1. ]]

Visualize single prediction:

Visualize multiple predictions:

These charts are largely unreadable, unlike the README example charts. The input data is all contrived samples just to get my workspace set up, so I've done virtually no customization aside from just installing stock packages.

Agnostic explanation without the model

Hi! Is there any way to make an agnostic explanation (get shap values) without the trained model? I mean, just use the train/test dataset and its expected/obtained output, no model involved in the process of the explanation.

SHAP Summary not returning feature value colors?

My SHAP summary plots are only displaying in red and it is not showing the feature value bar. I just upgraded shap to the newest package and also tried the '%config InlineBackend.figure_format = 'retina' because this added the feature value colors for me before.

Do you know what would cause this?

Here is how I am calling it -
shap_values = lgb_model.predict(X_validate, pred_contrib=True)
shap.summary_plot(shap_values, X_validate.columns)

X_validate is just a pandas dataframe from a train_test_split.

SHAP version -
Name: shap
Version: 0.12.0
Summary: A unified approach to explain the output of any machine learning model.
Home-page: http://github.com/slundberg/shap
Author: Scott Lundberg
Author-email: [email protected]
License: MIT
Location: /home/bbennett/miniconda3/envs/ml/lib/python3.6/site-packages
Requires: numpy, matplotlib, scipy, tqdm, scikit-learn, pandas, iml

Machine OS -
CentOS Linux release 7.4.1708 (Core)

Explanation plotting in python instead of JS

Hi @slundberg ,

Firstly, thank you for this great model interpretation algorithm and implementation! It's really amazing that you've also added it directly into the XGBoost and LightGBM models.

One question. I was trying to create a plot similar to the one you get from something like shap.visualize(shap_values[0,:], X.iloc[0,:]) without using a notebook. For example, for the dependency plot you are using matplotlib.pyplot.

Would it be possible to add an option for shap.visualize() to use matplotlib.pyplot instead of javascript (even if the new plot won't be interactive)? Or, could you give me some guidance for doing so?

IndexError: invalid index to scalar variable.

Traceback:
IndexError Traceback (most recent call last)
in ()
----> 1 shap.summary_plot(shap_values,z)

/usr/local/lib/python3.6/site-packages/shap/plots.py in summary_plot(shap_values, features, feature_names, max_display, plot_type, color, axis_color, title, alpha, show, sort, color_bar, auto_size_plot)
339 if sort:
340 # order features by the sum of their effect magnitudes
--> 341 feature_order = np.argsort(np.sum(np.abs(shap_values), axis=0)[:-1])
342 feature_order = feature_order[-min(max_display,len(feature_order)):]
343 else:

IndexError: invalid index to scalar variable.

Minor copy/paste errors in LGBM classification notebook

Hi,

I noticed that the description under the Explain Predictions header in the LGBM classification notebook was copied over from the Keras notebook and doesn't apply.

EDIT: Description also doesn't apply under the Visualize a single prediction header

how does the algorithm consider the number of subsets behind each path?

I'm very interested in your Shapely method for consistent feature attribution for trees (https://arxiv.org/pdf/1706.06060.pdf) , but can't understand what you are doing there no matter how much effort I'm giving it. The intuition is very clear to me, but the algorithm itself is indecipherable for me :(

Let's take a concrete example:
10 features named A,B,C...J
The leaf nodes are labeled X1, X2...
For the sample we want to explain, the hot path for every feature is always left.

     A
   /    \
  X1     B
       /   \
       X2   C
           /  \
           X3  X4

I'm interested in phi(B, X2) and phi(B, X3)

For phi(B, X2), the relevant paths are (A=0, B=0) - positive , (A=0, B=1) - negative.
There are 2^8 subsets of each.

For phi(B, X3), the relevant paths are (A=0, B=0, C=0) - negative, (A=0, B=0, C=1) - negative.
There are 2^7 subsets of each.

I don't understand in which line in the algorithm are you taking into account the 2^8 vs. the 2^7 subsets? (I'm simplifying and ignoring the fact that we should weigh each subset according to its total number of ones)

sorry if this sounds trivial to you or if I'm missing something fundamental, it's just that the intricacies of UNWIND and EXTEND left me really confused...

Thanks so much for your time, Ido

SHAP in unsupervised anomaly detection

Hi @slundberg ,

I'm facing a specific use case where I try to interpret multivariate anomaly detection algorithms. So, I have a dataset, I apply an unsupervised anomaly detection algorithm (e.g. Isolation Forest) and get which data samples are anomalous. Then, I build a supervised model (XGBoost) and I'm getting the SHAP values in order to explain the anomalies.

I have a question that is more or less 'philosophical'. Since I will not be using the supervised model to make predictions on new data, how should the model be trained? Is it ok to train the XGBoost model on the dataset and then make the predictions (and get the SHAP values) on the same dataset? Could the SHAP values 'overfit' since I'm using the model on the dataset it was trained on? Or should I, still, create different train-test datasets?

My guess is that I should still use different train/test sets but I'm not convinced this is the correct approach. Any thoughts?

Typos in the Tree SHAP arXiv paper

1. figure 1, left side talks about "married" and "age" features but is captioned with features "cough" and "fever"
2. Shapely formula in page 17 is wrong, M-|S|!-1 should be (M-|S|-1)!
3. indexing in EXTEND is broken? first time we call it l=len(m)+1=0+1=1, then we update m[l+1] which is m[2]. is that what you meant?

No module named 'shap.explainers' with 0.13 in LightGBM notebook

After installing 0.13 (with lightgbm 2.1.0), the import for the notebook 'Census income classification with LightGBM' fail as follows (works fine with 0.12.1):

ModuleNotFoundError                       Traceback (most recent call last)
<ipython-input-1-5f776e0a17f7> in <module>()
      1 from sklearn.model_selection import train_test_split
      2 import lightgbm as lgb
----> 3 import shap
      4 
      5 # print the JS visualization code to the notebook

~/.virtualenvs/shap/lib/python3.6/site-packages/shap-0.13-py3.6.egg/shap/__init__.py in <module>()
      5 from iml.links import Link, IdentityLink, LogitLink
      6 from iml.common import Instance, Model
----> 7 from .explainers.kernel import KernelExplainer, kmeans
      8 from .explainers.tree import TreeExplainer
      9 from .plots import visualize, plot, summary_plot, joint_plot, interaction_plot, dependence_plot, force_plot

ModuleNotFoundError: No module named 'shap.explainers'

pip list

backcall (0.1.0)
bleach (2.1.3)
cycler (0.10.0)
decorator (4.2.1)
entrypoints (0.2.3)
html5lib (1.0.1)
iml (0.5.1)
ipykernel (4.8.2)
ipython (6.3.1)
ipython-genutils (0.2.0)
ipywidgets (7.2.0)
jedi (0.11.1)
Jinja2 (2.10)
jsonschema (2.6.0)
jupyter (1.0.0)
jupyter-client (5.2.3)
jupyter-console (5.2.0)
jupyter-core (4.4.0)
kiwisolver (1.0.1)
lightgbm (2.1.0)
llvmlite (0.22.0)
MarkupSafe (1.0)
matplotlib (2.2.2)
mistune (0.8.3)
nbconvert (5.3.1)
nbformat (4.4.0)
notebook (5.4.1)
numba (0.37.0)
numpy (1.14.2)
pandas (0.22.0)
pandocfilters (1.4.2)
parso (0.1.1)
pexpect (4.4.0)
pickleshare (0.7.4)
pip (9.0.3)
prompt-toolkit (1.0.15)
ptyprocess (0.5.2)
Pygments (2.2.0)
pyparsing (2.2.0)
python-dateutil (2.7.2)
pytz (2018.3)
pyzmq (17.0.0)
qtconsole (4.3.1)
scikit-learn (0.19.1)
scipy (1.0.1)
Send2Trash (1.5.0)
setuptools (39.0.1)
shap (0.13)
simplegeneric (0.8.1)
six (1.11.0)
terminado (0.8.1)
testpath (0.3.1)
tornado (5.0.2)
tqdm (4.21.0)
traitlets (4.3.2)
wcwidth (0.1.7)
webencodings (0.5.1)
widgetsnbextension (3.2.0)

alternative `shap.visualize`

I think it looks awesome, but it did somewhat confuse me, e.g.:

• I got directions wrong. The things that increase the prediction are on the 'negative' side of the graph. (I think.)
• I wasn't sure if the arrows were overlapping or stacked. Either way, it's hard to compare multiple variables. (If it's interactive, then that's OK, but it's of less use for e.g. a printed copy, or in a PPT).

So possibly you could provide different plot styles, as per summary_plot. The obvious example is a horizontal bar chart, though you can do prettier things with e.g. scatters. That said, these will be less pretty when there are many features.

Just a thought - of course, we can always do our own. (I am quite happy with a printed list!)

KernelExplainer: Getting correct result with a small number of features

I used the KernelExplainer for a dataset with only three independent features. It then produces wrong results. According to Scott Lundberg the reason probably is that he has hardcoded BIC lasso feature selection as a preprocessing step in the code. This is likely failing for the small example by incorrectly not selecting all the features before doing the kernel estimate. Scott is working on adding a parameter to control feature selection (or turn it off) and meanwhile he wanted me to open a github issue for this task.

Should I use SHAP contributions for debugging why did my XGB classifier does not transfer to another population?

I trained a tree-ensemble classifier (XGBOOST) on population A, validated it and I'm satisfied with its accuracy (AUC 0.78). Now I'm trying to transfer it to a slightly different population B, and there the accuracy of the model deteriorates badly (AUC 0.68)

I tried isolating which of the features did not transfer well, both by simple univariate analysis (comparing distributions), and by comparing each feature correlation with the label, and couldn't find anything obvious.

I thought about comparing the label distributions at every node in every tree for the validation populations in A and B, thereby testing all conditional probabilities the model assumes are holding actually hold at B.

Could this be achieved by comparing the distributions of the feature contributions in A and B? If I'm looking for a bug in the specific classifier I trained, should I use SHAP for or the original "approximate tree-path contributions", as they are directly associated with what the classifier is actually doing (while SHAP contributions are designed to give the 'fair' contributions for each feature, not for debugging a specific classifier)?

Interpretation of Kernel SHAP and its hyperparameters

Hi,

I have the following questions with regards to KernelExplainer:

1. I am setting explainer = KernelExplainer(predictor, background, nsamples) I observe the algorithm is slow even if I set nsamples=1 when the background data set is large. Let's say I have a data set of size n in the background, and set nsamples=k. Each sample is of dimension d. Can I know how many evaluations of predictor are made in total under this setting?

2. I am interested in the definition of the map x' -> x, and how the algorithm is implemented when we have real valued x. I have the following interpretation of the algorithm:
   1.Input a sample x, a model f, and the background data set {x1,x2,...,xN} . We want to interpret x.
   2.Replace a subset of entries in x, by entries in xi, where xi is randomly sampled from {x1,x2,...,xN}. Evaluate the function after replacement.
   3.Repeat the above for different subset of entries (dimensions), chosen according to the Shapley kernel.
   4.Compute the Shapley value approximation.

3. Also, It seems a single reference point is assumed in the paper, but not in the implemented algorithm. Does 'background' serve as the function of multiple reference points?

Thanks!

TypeError: predict() got an unexpected keyword argument 'pred_contribs'

Hi everyone,

I'm using xgboost 0.6 (on Anaconda, python 2.7). When trying to run sample of code from the documentation (with boston dataset) I get an error - TypeError: predict() got an unexpected keyword argument 'pred_contribs'
Could you advise please?

Thank you!

Does this work with multi-class classification (a Keras NN model)

I'm following your notebook with the NN example. My model output is a soft-max over some number of classes. I define the f in your notebook as below:

def f(X):
    return model.predict(X)

The call to explainer.explain succeeds. However, the call to visualize fails, with the error:

/usr/local/lib/python2.7/dist-packages/iml/visualizers.pyc in __init__(self, e)
    110         # build the json data
    111         features = {}
--> 112         for i in filter(lambda j: e.effects[j] != 0, range(len(e.data.group_names))):
    113             features[i] = {
    114                 "effect": ensure_not_numpy(e.effects[i]),

ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()

A simple fix?

Interaction contribution in lightgbm

Wondering if the interaction contribution calculation available in Xgboost is possible with lightgbm also. Had a quick look at the source but did not find anything.

Thanks for the great library by the way!

Anyway you could update using this in python using XGBoost?

Was trying to use the examples you provided but can't seem to get it working with the current version of XGBoost.

Treeinterpreter

Hi
I have the question that might seem naïve
Lets say i want to use the part of shap where we use the treeinterpreter ..is that possible?
I know this is a unified approach , but lets say if i want aonly to test how it works by adding or removing different methods?
Like if i want to remove the lime part and test how this will work unifying the other approaches (deeplift, treeinterpreter

use “shap.force_plot()”, then "MemoryError"，maybe need "gc.collect()"

run shap.force_plot()
result MemoryError:

Int64Index: 213118 entries,
Columns: 109 entries,
dtypes: float64(40), int64(33), uint8(36)
memory usage: 137.6 MB

Running out of memory when testing segmented images

I'm using the ImageNet VGG16 Model with Keras notebook as a guide to test my own TensorFlow model and set of images.

The problem is that I run out of memory when doing:
shap_values = explainer.shap_values(np.ones((1,50)), nsamples=1000)

I can fit only 10 samples in memory which is not enough to generate the SHAP values for all 50 segmentations. Is there a way to call the SHAP explainer on each image segment permutation individually, instead of fitting them all in one batch?

One idea I had is:

shap_values <- zeros(n_segments)
while minimum(shap_values) is 0
    new_values <- explainer.shap_values(np.ones((1,50)), nsamples=small_number)
    shap_values <- element-wise_maximum(shap_values, new_values)

However, this results in a lot of redundant computation and I'm not sure if SHAP values from different runs are directly comparable like this.

Base values always conditional?

Hi,

I've noticed that all the sample notebooks appear to require training data to ascertain base values. But say all I've got is a black box model. I don't know its architecture and I don't have access to the original training data. Will I simply have to guess E[f(x)] from my test data? Is it fair to say that base value estimates are always conditional on some input, and that only the training data used to fit the model is guaranteed to provide unbiased estimates of E[f(x)]?

Thanks,
David

One hot encoded example

How would the Census income classification with Keras.ipynb example work if the features were one hot encoded ? I spent some time on trying to figure this out yesterday but couldn't get it to work any ideas would be really appreciated.

Thanks for a great paper and library this looks really useful.

Multiclass example & MNIST

HI,

Do you happen to have an example for the multi-class setting? In your paper, you had an example on MNIST, but couldn't find an MNIST example as one of your notebooks. The key question here is whether to explain each output dimension separately or in a combined manner?

Thanks!

Output value in binary classification task is outside [0, 1] range

Hi @slundberg,

I've been playing with a binary classification task using XGBoost and I noticed an unexpected (for me at least) behaviour. I replicated it using the adult dataset you're providing.

So, after training a binary classfication XGBoost model and plotting the SHAP values for a case, I'm getting the following:

Both the base value and the output value are outside the [0, 1] range. Is this the expected bahavior? If so, how can someone interpret this?

Output to file

Currently, the SHAP module uses a mix of matplotlib and IML to visualize the prediction explanations. It would be great if there was an easy way to save these to a file. For the matplotlib charts, that is easy enough to just call savefig, but saving the IML output is almost impossible.

Is there any way we can achieve this without having to make changes deep within the IML code?

Shapley values for variable importance?

Hi,

I'm curious if there is some way to use Shapley values to estimate global variable importance metrics for a model? Say I use dataset X to train classifier M, then run shap on every observation in X. Is there some principled way to extract variable importance measures from the resulting Shapley matrix? I could imagine a few potential options, e.g. featurewise means of absolute values or featurewise variances, but these all seem a little ad hoc. Perhaps the SVD of the Shapley matrix may be informative here?

Thanks,
David

Exception in KernelExplainer.solve()

Hi,
Thanks for you work. I have used DeepLift and I am trying out this approach.

I have a feed forward neural network built using Keras. All features in my dataset are continuous. There are 31 features. I setup up the python program just like the KNN example on github landing page.

I get the following error. Let me know if I am doing anything wrong that is triggering this.

Thanks
Param

Error:
IndexError Traceback (most recent call last)
in ()
26 print(x.shape)
27 print(x)
---> 28 visualize(explainer.explain(x))

/Library/Python/2.7/site-packages/shap/shap.pyc in explain(self, incoming_instance, **kwargs)
169
170 # solve then expand the feature importance (Shapley value) vector to contain the non-varying features
--> 171 vphi,vphi_var = self.solve(self.nsamples/self.max_samples)
172 phi = np.zeros(len(self.data.groups))
173 phi[self.varyingInds] = vphi

/Library/Python/2.7/site-packages/shap/shap.pyc in solve(self, fraction_evaluated)
324 #nonzero_inds = np.arange(self.M)
325
--> 326 eyAdj2 = eyAdj - self.maskMatrix[:,nonzero_inds[-1]]*(self.link.f(self.fx) - self.link.f(self.fnull))
327 etmp = np.transpose(np.transpose(self.maskMatrix[:,nonzero_inds[:-1]]) - self.maskMatrix[:,nonzero_inds[-1]])
328 #print(self.maskMatrix)

IndexError: index -1 is out of bounds for axis 0 with size 0

Algorithm explanation

@slundberg Could you please provide an explanation of how the space of simplified inputs described in SHAP paper is constructed in your algorithm for the x given? Where is h_x mapping introduced in the code?
Your help would be very appreciated.

Tree SHAP for random forests?

Hi,

Tree SHAP seems to work great on boosted tree models like XGBoost. But after reading the paper on Consistent feature attribution for tree ensembles I'm wondering if there's some reason the algorithm couldn't be applied to other tree-based ensemble methods like random forests? Implementing this functionality in python or R for arbitrary tree-based models could be extremely useful! But maybe tree SHAP is tailored to boosting and unfit for bagging in some way I'm failing to appreciate?

Thanks,
David

null's processing in XGBoost

Hi!

Do you know how SHAP walks in XGBoost trees during Shap values calculation? You know there is algorithm of Shap values calculation which says: in case of NULL you should follow both children and there is XGBoost NULL processing logic which says: you should go to only one more optimal child node.

Please clarify.

Sergey.

is summary plot only for classification?

Thank you for this amazing work. Just wondering, I want to identify variable importance using the summary plot. But my model is a tree-based regressor. I am not sure if I understand the paper correctly, I found all examples calculating shap values are all classifications. Could you please help clarify this, can this be used in regression? Thank you so much!

Support summary_plot for categorical features

We have implemented shap_values in catboost (http://catboost.yandex or https://github.com/catboost/catboost), it works really cool, thank you very much!
It'll be on pypi in tomorrow's release.

One problem with visualization is that you cannot visualize summary_plot if some features are categorical.
I suggest in this case to draw all the dots in gray color for example. It will still make a lot of sense, because you see the effect of categorical feature value on the objects in the train dataset.

Here is a code example that fails:

X_adult,y_adult = shap.datasets.adult(display=True)
cat_features = [1, #Work
                    3,#Marital status
                    4,#Occupation
                    5,#Relationship
                    6,#Race
                    7,#Sex
                    11 #Country
]
from catboost import *
model = CatBoostClassifier(
    iterations=300, learning_rate=0.1, random_seed=123,
    max_ctr_complexity=1, # Currently we don't calculate shap values for models with categorical feature combinations.
                          # This will be changed in one of the next releases.
)
model.fit(X_adult, y_adult, cat_features=cat_features, verbose=False)
shap_values = model.get_feature_importance(data=Pool(X_adult, y_adult, cat_features=cat_features), fstr_type='ShapValues')
shap.summary_plot(shap_values, X_adult)

We have also added a tutorial to our github, it uses a summary_plot for cat features, and it's informative (are you ok with us taking the first part from Shap values?): https://github.com/catboost/catboost/blob/master/catboost/tutorials/advanced_tutorials/shap_values_tutorial.ipynb

SHAP for text classification with CNN or RNN

Hi,

We are thinking of calculating shap values for text classification models like CNN or RNN. I'm just wondering whether we can utilize shap values in this problem setting.

The model takes tokenized sequence of integers, each of which corresponds to one word. If I understand correctly, this package assumes feature independence, but it is reasonable not to assume this here. Rather, we believe that n-grams play a key role in classification.

I guess we can get shap values within a reasonable time if only one reference value (like median value) is passed to KernelExplainer, but do you think it captures some importance of each feature in this setting?
If not, do you have any ideas to get some sort of interpretablity of the trained model in this setting?

Thanks in advance.

pip install error

I'm not sure why, but I couldn't install from pip.
I try below command

$ pip install --user shap
shap/_cext.cc:5:10: faltal error: tree_shap.h: No such file or directory
     #include "tree_shap.h"

I was able to install from setup.py

$ python setup.py install --user

dependence_plot: scale axis, increase size, ...

Hello,

it's not in documentation, and not in the example, but can we play somehow with the dependence_plot?

Thank you for this wonderful tool!

force_plot for single sample gets cut off

I'm following the Census income classification with scikit-learn notebook(https://slundberg.github.io/shap/notebooks/Census+income+classification+with+scikit-learn.html) and when I try to plot the shap_values for a single sample in the notebook, I can't get the full plot out.

Here is the output:

Surprisingly, if I try to plot many samples there is no problem. See output:

I'm using python3, pip installed shap (0.12.0) in a jupyter notebook on windows.

I don't understand the simple example for inconsistent feature attribution in the paper

even if I ignore the wrong labels in the left tree, still the expected value in the root is:
100 * 25% + 0 * 25% + 0 * 25% + 0 * 25% = 25
and cough=Y changes it to:
100 * 50% + 0 * 50% = 50
so the output task should assign it 25, and not 50 as written...

thanks again for your patience, Ido

shap.visualize issue

I can't seem to get shap.visualize to work for the second plot on the readme. I upgraded shap and xgboost.

This is the code I'm using and the error. Any help would be great.

import xgboost
# or import lightgbm
import sklearn.datasets
import shap

# load JS visualization code to notebook
shap.initjs() 

# train XGBoost or LightGBM model
d = sklearn.datasets.load_boston()
bst = xgboost.train({"learning_rate": 0.01}, xgboost.DMatrix(d.data, label=d.target), 100)
# or bst = lightgbm.train({"learning_rate": 0.01}, lightgbm.Dataset(d.data, label=d.target), 10)

# explain the model's prediction using SHAP values on the first 1000 training data samples
shap_values = bst.predict(xgboost.DMatrix(d.data), pred_contribs=True)
# or shap_values = bst.predict(d.data[0:1000,:], pred_contrib=True)

# visualize the first prediction's explaination
shap.visualize(shap_values[0,:], feature_names=d.feature_names, data=d.data[0,:])

# visualize the training set predictions
shap.visualize(shap_values, d.data)
---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
~\AppData\Local\Continuum\Anaconda3\lib\site-packages\IPython\core\formatters.py in __call__(self, obj)
    339                 pass
    340             else:
--> 341                 return printer(obj)
    342             # Finally look for special method names
    343             method = get_real_method(obj, self.print_method)

~\AppData\Local\Continuum\Anaconda3\lib\site-packages\iml\visualizers.py in try_list_display(e)
     66     def try_list_display(e):
     67         if isinstance(e, collections.Sequence) and len(e) > 0 and isinstance(e[0], AdditiveExplanation):
---> 68             return visualize(e).data
     69         else:
     70             return str(e) if old_list_formatter is None else old_list_formatter(e)

~\AppData\Local\Continuum\Anaconda3\lib\site-packages\iml\visualizers.py in visualize(e)
     54         return SimpleListVisualizer(e).html()
     55     elif isinstance(e, collections.Sequence) and len(e) > 0 and isinstance(e[0], AdditiveExplanation):
---> 56         return AdditiveForceArrayVisualizer(e).html()
     57     else:
     58         assert False, "visualize() can only display Explanation objects (or arrays of them)!"

~\AppData\Local\Continuum\Anaconda3\lib\site-packages\iml\visualizers.py in __init__(self, arr)
    169                 "features": {}
    170             })
--> 171             for i in filter(lambda j: e.effects[j] != 0 or e.instance.x[0,j] != 0, range(len(e.data.group_names))):
    172                 self.data["explanations"][-1]["features"][i] = {
    173                     "effect": ensure_not_numpy(e.effects[i]),

~\AppData\Local\Continuum\Anaconda3\lib\site-packages\iml\visualizers.py in <lambda>(j)
    169                 "features": {}
    170             })
--> 171             for i in filter(lambda j: e.effects[j] != 0 or e.instance.x[0,j] != 0, range(len(e.data.group_names))):
    172                 self.data["explanations"][-1]["features"][i] = {
    173                     "effect": ensure_not_numpy(e.effects[i]),

IndexError: index 13 is out of bounds for axis 0 with size 13

list index out of range when using summary_plot

I'm using XGBoost for a classification problem (latest version 0.7). As parameter for XGBoost I'm setting param['num_class'] = 5 as I have 5 classes. My data has 25 features. I use XGBoost to get the shap values by doing: bst.predict(xg_test, pred_contribs=True). From XGBoost's documentation I can conclude that n_features + 1 shap values should be returned (per sample). However what is being returned is (n_features+1) * num_class (which is 130 for my problem. Because of this, summary_plot gives an error as I provide it with 25 feature names.

It this intended? Or should one for example select the range of 25 features from the 130 features that correspond to the correct class? E.g.: shap_values[:,:len(xg_test.feature_names)] for the first 25 values.

(Paper): Conditional Expectation Calculation & Inverse Function Questions

Note: these questions refer to the paper underlying this method, not directly to the code

1. In equation 12, you argue that you can calculate the conditional expectation term by simply calculating the function value using the actual instance values of the features in set S, and the expected value of the features in set S-complement (essentially, it seems, marginalizing over the features in set S-complement). But, then, in the kernel approach, it's not clear to me how you're calculating this expectation. Is the idea that, for all features in z' that are in set S-complement (that is to say: are set to 0) their corresponding values in the non-simplified vector are set to the overall average (i.e. expected value) of each feature? That seems to be the implication, but I can't find anywhere in the paper where it definitively says so.

2. In Theorem 2, why is an inverse function of h_x applied to the z' vector? In all prior cases, the mapping function h_x has taken an input of simplified features, and given an output of non-simplified features, suitable for being given to your trained model. Why, in this case, would we be using an inverse of that function when the input we're giving it is a z' simplified feature vector?

Distribution in summary plot

Interested in me submitting a PR for this? We've used it internally to make it more interpretable. If the lowest kde subplot is for variable X, then dark blue means 'bottom 20% of X values' up to red which is 'top 20% of X values'. We find this more interpretable as e.g. in this case we can quickly see that low values of X tend to decrease SHAP values, while high values increase it, etc. Otherwise it was hard to explain to clients, especially for unbalanced kdes like the middle one (which might be 'the wrong way round').

Notebook examples

Hi,

First of all, thank you for this interesting algorithm and code!

I tried running the notebooks you provided, but it seems that some do not match the current version of the code (probably due to improvements you made the last 6 months).

I tried Diabetes regression.ipynb which had for example a problem with the function ShapExplainer that does not exist anymore (probably now KernelExplainer from shap.explainer). I assume there are more issues like this for other notebooks.
I also tried a notebook updated 3 weeks ago which works fine.
Could you commit an update of the notebooks? These are very useful to understand how we should/could use your code!

thanks,

Integration with React

I was reading the comments on other issues, and noticed that the visualizations are made with D3.js and React. I was wondering if there's a way to render them directly in a React front-end.

For example, a request is made from a React front-end for the explanation of a prediction. This calls a REST-API on the server, which returns the SHAP values, model feature names, and the data used to create the prediction (basically the shap.visualize() parameters). Finally, on React, the same visualization created on the notebooks is displayed on the app.

One alternative to this is creating an image from the visualization, in the case of single explanations, like the one mentioned on #3 . But a native result would be preferable. ¿Is there currently a way to do this?

Thanks in advance!

A Spark version in plan?

Thanks for the great package. Is there a plan to implement/integrate the SHAP method in PySpark?

Of, if possible, what would be your suggestion to use this technique in Spark?

thanks,
Richard

Possible to include Anchor?

A new method was released by the authors of LIME. Would there be any value in adding this somehow to the SHAP package?

Paper title: "Anchors: High-Precision Model-Agnostic Explanations"
Paper: https://homes.cs.washington.edu/~marcotcr/aaai18.pdf
Source: https://github.com/marcotcr/anchor

Adjusting number of features shown

When visualizing a single prediction, it appears as though the IML visualizer decides how many feature names to put in the chart. It would be great if we had more control over the features shown. For instance, showing only the top-n features for both the positive and negative direction.

Is it possible to achieve this in the code as it stands or do it require deep changes to IML?