zju-vipa / transferbilityfromattributionmaps

(NeurIPS 2019) Deep Model Transferbility from Attribution Maps

Deep Model Transferbility from Attribution Maps

• "Paper: Deep Model Transferbility from Attribution Maps", NeurIPS 2019.(released soon)

  J. Song, Y. Chen, X. Wang, C. Shen, M. Song

  Homepage of VIPA Group, Zhejiang University, China

Getting Started

These instructions below will get you a copy of the project up and running on your local machine for development and testing purposes.

Prerequisites

Install the following:

- Python >= 3.6
- Tensorflow >= 1.10.0
- Matlab R2019a

Then, install python packages:

pip install -r requirements.txt

In order to generate attribution maps from Deep Models, you also need to download DeepExplain which this project utilizes to generate attribution maps, and copy it to your project directory $DIR.

cp -r DeepExplain-master/deepexplain $DIR/lib/

Probe datasets

Those datasets involved in this project are:

• Taskonomy Tiny
• MS COCO Val 2014
• Indoor Scene

Make sure to download them and move to $DIR/dataset.

Those datasets need to be arranged in the following format:

|- dataset
|   |---taskonomy
|   |   |---collierville_rgb
|   |   |   |---point_0_view_0_domain_rgb.png
|   |   |   |---...
|   |   |---corozal_rgb
|   |   |---darden_rgb
|   |   |---markleeville_rgb
|   |   |---wiconisco_rgb
|   |---coco
|   |   |---COCO_val2014_000000000042.jpg
|   |   |---...
|   |---indoor
|   |   |---Images
|   |   |   |---airport_inside
|   |   |   |   |---airport_inside_0001.jpg
|   |   |   |---bowling
|   |   |   |---...

You can also check $DIR/explain_result/name_of_dataset/imlist.txt to find out how those images are arranged(Note that images in the imlist.txt are randomly selected from dataset, please feel free to test using more images).

Pre-trained Models

Download pre-trained models:

sh tools/download_model.sh

Running the tests

First, generate Attribution Maps and save corresponding Attribution maps to explain results directory:

cd tools
python deep_attribution.py --explain_result_root explain_result --dataset taskonomy --imlist-size 1000
python deep_attribution.py --explain_result_root explain_result --dataset coco --imlist-size 1000
python deep_attribution.py --explain_result_root explain_result --dataset indoor --imlist-size 1005

Calculate affinity matrix of those tasks according to the Attribution Maps:

python affinity.py --dataset taskonomy --imlist-size 1000
python affinity.py --dataset coco --imlist-size 1000
python affinity.py --dataset indoor --imlist-size 1005

Visualization

Plot P@K, R@K Curve, make sure you've already completed testing on three datasets:

python plot.py --fig-save fig

e.g.

Plot Task Similarity Tree:

matlab -nosplash -nodesktop task_similarity_tree.m

e.g.

Todo

Release code of experiment generating attribution maps based on other kinds of pre-trained models(different tasks, different architectures, etc.).

Apply more explain methods and see how they work.

Acknowledgement

This repo is built upon the code from Taskonomy.

And we thank deepexplain for providing the attribution tools.

Citation

If you find this code useful, please cite the following:

@inproceedings{ModelTransfer_NeurIPS2019,
  title={Deep Model Transferbility from Attribution Maps},
  author={Jie Song, Yixin Chen, Xinchao Wang, Chengchao Shen, Mingli Song},
  booktitle={NeurIPS},
  year={2019}
}

Contact

If you have any question, please feel free to contact

Jie Song, [email protected];

Yixin Chen, [email protected].

Homepage of VIPA Group, Zhejiang University, China