Question about the CLIP Demo about transformer-mm-explainability

[hila-chefer]

Hi @g-luo, thanks for your interest in our work!

• yes it is the same method. The difference stems from the different input to each model. ViT takes an image as input and each token is an image patch, therefore the output is a heatmap on the different patches. On the other hand, LXMERT takes as input bounding boxes and not image patches, so the output is the importance of each box, therefore we chose to present it in black and white shades and you can see that the important regions are actually bounding boxes.
  As for the text- I didn’t implement the text explainability, but I’ll be sure to do that to add textual explanations to CLIP as well for completeness.
• For CLIP with ResNet I’d suggest using a CNN explainability method. The easiest is GradCAM I think, which should also be class specific.

I hope this helps.

[g-luo]

Thanks so much for the clarifications @hila-chefer!