jebc314 / cpsc471finalproject

My CPSC 471 final project code for reproducibility.

CPSC471FinalProject

My CPSC 471 final project code for reproducibility.

Setup

Refer to the Conda environment configuration file environment.yml for the libraries required for running experiments.

In order to run process_data.ipynb, you need to download the ImageNet S-50 annotation dataset into a folder called "input\ImageNetS50". You also need to download the ILSVRC2012_img_train/(CLS).tar files into a folder called "input\images". After running process_data.ipynb, the output folders "input\images\(CLS)" will have the images correponding to the annotations in "input\ImageNetS50".

For recreating the paper's tests on their aggregation functions, you need to create an "ImageNet" folder. Then, running paper_replication.ipynb will produce "ImageNet/proper_data.pt", "ImageNet/preds.pt", "ImageNet/attributions_*.pt" for each of the attributions, and "ImageNet/agg.pt", which is the file with the aggregated explanations. To get the metric test results, you need to first create the ground truth masks using "ground_truth_mask.py". Then, create a "Results" folder and run faithfulness.py, randomization.py, robustness_file.py, sparseness.py, and localization.py to produce the corresponding metric scores.

Using the command "python (FILE) ImageNet/(AGGREGATED EXPLANATION).pt ImageNet" will produce the metric results. Refer to run_tests.ps1 for an example PowerShell script that produces metric results.

Aggregation Functions Experiments

For creating the aggregated explanations with the new aggregation functions refer to the second multiline comment in aggregation.py. Running python aggregation.py outputs the aggregated explanations in the "ImageNet" folder. Note that you should give your aggregate explanations file an unique name.

The metric_tests.ipynb file will produce the weighted and threshold aggregated explanations.

Use the same process as in setup to get the metric test results.

Pre-Aggregation Modifications

For generating the aggregation of a subset of explanations (gradient-based versus perturbation based), refer to the first multiline comment in aggregation.py.

In terms of the hyperparameter tuning experiments, the hyperparameter_tuning.py will produce the optimal hyperparameters for each the explainers. Note you have to create a "HyperparameterSearch" folder before running.

Run hyperparameter_analysis.ipynb to get the aggregated explanation. The caveat is that the individual attribution names have to be changed to the respective optimal one from runnning hyperparameter because they are timestamped.

For getting the attributions with the base model LIME uses changed, run improved_LIME.ipynb, and the aggregated explanations will be saved in the "ImageNet" folder.

Use the same process as in setup to get the metric test results.

Scope Expansion

For the CIFAR10 and MNIST datasets, the respective notebooks paper_replication_cifar10.ipynb (up till the section Ground Truth Masks can be run directly) and paper_replication_MNIST.ipynb will produce the needed files, including the transformed inputs, predictions, superpixel masks, and groud truth masks. For CIFAR10's ground truth masks, these sections had more manual changes because of the inconsistent results, which can make it more challenging to run. Refer to "CIFAR10/ground_truth_masks.pt" for the ground truth masks, since some masks were created by hand. The masks can be visually verified by plotting using Matplotlib's imshow. Refer to ground_truth_mask_tool.py for the tool that I used to make some masks manually. This tool needs the folder "CIFAR10/data/" with the images to mask and the folder "CIFAR10/masks/" to store the masks. The "N" button moves to the next picture, and the "S" button saves the mask.

For running the aggregation process on a vision trasformer, you need to create the folder "ImageNet_VT". The VT_generate_preds.py, VT_generate_expanations.py, and VT_aggregation.py will generate the aggregated explanations. The powershell run_VT_pars.ps1 has an example script to generate explanations using a subset of the explainers. The file ground_truth_mask_VT.py will generate the ground truth masks.

Use the same process as in setup to get the metric test results. Just make sure to change the folder name to the respective tests, like ImageNet to MNIST or to ImageNet_VT.

The files randomization_eMPRT.py and localization_RMA.py will evalute an explanation according to the respective metrics. Refer to run_tests_metric.ps1 for an example run evaluating some aggregated explanations.