Can the attribution values be visualized in Tensorboard mode? about what-if-tool

[jameswex]

Thanks for reaching out Tanmay and glad you were able to get attributions working with WIT in notebook mode.
Unfortunately, WIT in TensorBoard doesn't currently support attributions. This is because WIT in TensorBoard queries the models through TF Serving's PredictionService API, which only returns prediction results from the served model, and not other custom calculated values such as shap values. We rely on the model being queried to provide the attribution results (just as you did with a custom prediction fn when using attributions in notebook mode).
When you mention wanting to use TB mode due to a bigger data size, what exactly do you refer to? Are you able to load more data points when using WIT in TB vs in notebook mode, or were you just experimenting with the limits of each mode?

[tanmayg]

Thanks a lot for replying and clarifying James, really appreciate your prompt response.
When I say bigger data size, I meant a test dataset of 100000 data points and ~2000 features. In the notebook mode, I experimented with a subset of my dataset of 500 data points and 50 features and was able to get the desired results. But when I triggered the WIT tool with complete dataset, the kernel kept dying repeatedly, so I thought it might be memory issue for notebook and thus wanted to experiment with the dataset in Tensorboard mode.
Please suggest for better ways to handle bigger sets for WIT, if possible. Regards.

[jameswex]

WIT does have limits on the amount of datapoints it can handle, which changes based on the size of each datapoint. This is because WIT sends all datapoints to the frontend for interactive analysis.

I'm curious how many data points you are able to use it with if you keep your same number of features but decrease the number of datapoints, and vice versa, if you are able to provide that data.

In general, for large-scale analysis, I would suggest tools that do offline processing, such as TensorFlow Model Analysis (https://www.tensorflow.org/tfx/model_analysis/install). That tool can be used in conjunction with WIT in colab/jupyter, where you find interesting data slices with TFMA, and then can load samples from that slice in WIT for further analysis. See https://ai.googleblog.com/2019/12/fairness-indicators-scalable.html for details on using TFMA's FairnessIndicators together with WIT.

[jameswex]

WIT does have limits on the amount of datapoints it can handle, which changes based on the size of each datapoint. This is because WIT sends all datapoints to the frontend for interactive analysis.

I'm curious how many data points you are able to use it with if you keep your same number of features but decrease the number of datapoints, and vice versa, if you are able to provide that data.

In general, for large-scale analysis, I would suggest tools that do offline processing, such as TensorFlow Model Analysis (https://www.tensorflow.org/tfx/model_analysis/install). That tool can be used in conjunction with WIT in colab/jupyter, where you find interesting data slices with TFMA, and then can load samples from that slice in WIT for further analysis. See https://ai.googleblog.com/2019/12/fairness-indicators-scalable.html for details on using TFMA's FairnessIndicators together with WIT.

[tanmayg]

Thanks again James for your inputs. Really helped me a lot. I'm closing the issue now as I have a direction to go and experiment. I have another concern w.r.t 3d inputs in google explain for which I would open a new issue.