kiraving / seggradcam

SEG-GRAD-CAM: Interpretable Semantic Segmentation via Gradient-Weighted Class Activation Mapping

SEG-GRAD-CAM

Publicly available implementation in Keras of our paper "Towards Interpretable Semantic Segmentation via Gradient-Weighted Class Activation Mapping" by Kira Vinogradova, Alexandr Dibrov, Gene Myers.

Check out our poster for a schematic overview of the method.

No updates or upgrades to newer versions are planned.

Installation

pip install git+https://github.com/kiraving/SegGradCAM.git

Requirements

Python 3.6, (recommended) Anaconda, versions of other packages can be found here

Please download Cityscapes (Fine annotations) if you intend to test Seg-Grad-CAM on a real-world dataset collected on German roads.

Usage

• Code for Seg-Grad-CAM method
• Notebook for training, loading pretrained model and usage of Seg-Grad-CAM on TextureMNIST
• Training a U-Net with a backbone on Cityscapes & applying Seg-Grad-CAM
• Vanilla U-Net on Cityscapes & Seg-Grad-CAM

Credits:

CSBDeep

@article{weigert2018content,
  title={Content-aware image restoration: pushing the limits of fluorescence microscopy},
  author={Weigert, Martin and Schmidt, Uwe and Boothe, Tobias and M{\"u}ller, Andreas and Dibrov, Alexandr and Jain, Akanksha and Wilhelm, Benjamin and Schmidt, Deborah and Broaddus, Coleman and Culley, Si{\^a}n and others},
  journal={Nature methods},
  volume={15},
  number={12},
  pages={1090--1097},
  year={2018},
  publisher={Nature Publishing Group}
}

Cityscapes dataset:

@inproceedings{Cordts2016Cityscapes,
title={The Cityscapes Dataset for Semantic Urban Scene Understanding},
author={Cordts, Marius and Omran, Mohamed and Ramos, Sebastian and Rehfeld, Timo and Enzweiler, Markus and Benenson, Rodrigo and Franke, Uwe and Roth, Stefan and Schiele, Bernt},
booktitle={Proc. of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
year={2016}
}

segmentation_models package:

 @misc{Yakubovskiy:2019,
    Author = {Pavel Yakubovskiy},
    Title = {Segmentation Models},
    Year = {2019},
    Publisher = {GitHub},
    Journal = {GitHub repository},
    Howpublished = {\url{https://github.com/qubvel/segmentation_models}}
  }

TextureMNIST dataset Code for toy dataset generation of "Grid Saliency for Context Explanations of Semantic Segmentation" paper

How to cite Seg-Grad-CAM:

@inproceedings{Vinogradova2020TowardsIS,
  title={Towards Interpretable Semantic Segmentation via Gradient-weighted Class Activation Mapping},
  author={Kira Vinogradova and Alexandr Dibrov and Eugene W. Myers},
  booktitle = {Proceedings of the AAAI Conference on Artificial Intelligence},
  year      = {2020},
  doi       = {10.1609/aaai.v34i10.7244}
}

seggradcam's Issues

Tensorflow2

Hi,

Do you have any plans to update this repository to Tensorflow 2? I am working with your SegGradCAM code for explainable AI, but my modifications to allow for use of Tensorflow 2 have not been successful yet (since my model is built in Tensorflow 2). So far I have found the main difference to be how the gradients are computed (GradientTape() vs gradient()). Thanks!

gradcam and segmentation output

hello author,

I have a question related to grad cam. I want to know the difference between class activation maps and output feature map. If i apply the gradcam module for the last conv block or intermediate conv blocks, it is supposed to give the heatmap about where in the image the channels are activated. In case of classification task it maskes sense. What about in case of segmentation task?
If i apply grad cam to the last conv block, the gradcam heatmap is same as the output feature map or output segmentation output?
I have tried this on my task, and i found that the gradcam heatmap and final output is different. What can i conclude from this?

thank you

Support for Tensorflow 2.X

The codes throw error for Tensorflow 2.X versions. Are there any plans of updating the codes to support TF2.X versions? Thanks.

ask for a pytorch version

Thanks for your great work, could you provide a pytorch version of seg-grad-cam？

it doesn't import in Google Colab

Hello dear Kira
I'm working on your seg_grad cam code but unfortunately when I want to import Seggradcam libraries it doesn't work, I face this error:
`---------------------------------------------------------------------------
ImportError Traceback (most recent call last)
in ()
----> 1 import seggradcam.metrics

3 frames
/usr/local/lib/python3.7/dist-packages/keras/backend.py in ()
35 from tensorflow.python.distribute import distribute_coordinator as dc
36 from tensorflow.python.distribute import distribute_coordinator_context as dc_context
---> 37 from tensorflow.python.eager.context import get_config
38 from tensorflow.python.framework import config
39 from keras import backend_config

ImportError: cannot import name 'get_config' from 'tensorflow.python.eager.context' (/usr/local/lib/python3.7/dist-packages/tensorflow/python/eager/context.py)

---

NOTE: If your import is failing due to a missing package, you can
manually install dependencies using either !pip or !apt.

To view examples of installing some common dependencies, click the
"Open Examples" button below.
---------------------------------------------------------------------------`
DO you have any idea?
Thanks

Pink Outline

Hi,

Is it possible to remove the pink outlines in the gradcam image. Can you please let me know which codes to modify to remove the outlines.

Thank you very much

IndexError: index 128 is out of bounds for axis 0 with size 128

Hello,

When I run SegGradCam code to use GradCam in Thresholding Segmentation with PixelRoI,
an unexpected error occurs:

Code:

from imageio import imread
image = imread("../input/retinal-vessel-segmentation/DRIVE/training/images/21_training.tif")
image = tf.image.resize(images=image, size=[128, 128])
image = image.numpy()
image = image.astype(np.uint8)
data = Image.fromarray(image)
data.save('dummy_pic.jpg')

a = "./dummy_pic.jpg"
a = imread(a)

roi=PixelRoI(128,128,a)

Error:

---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
/tmp/ipykernel_35/845604040.py in <module>
      1 # image = imread(images[0], 3)
----> 2 roi=PixelRoI(128,128,a)
      3 # roi.shape()

/tmp/ipykernel_35/2143735659.py in __init__(self, i, j, image)
     64         self.image = image
     65         self.roi = np.zeros((image.shape[-3], image.shape[-2]))
---> 66         self.roi[i, j] = 1
     67         self.i = i
     68         self.j = j

IndexError: index 128 is out of bounds for axis 0 with size 128

Environment:

Desktop : Virtual Environment Kaggle, GPU
Tensorflow : 2.6.2 (unupgradable)
Matplotib : 3.5.1 (unupgradable)

Question:

Can anyone please help me to solve this error?