seth-park / multimodalexplanations Goto Github PK

Hi. I've been trying to train my own models for vqa-x, but when I try to generate explanations on models I trained using train.py, my vqa answers and explanations are horrible.

I train for 50k iterations and the print results during training look great, but when I run generate_explanations.py, I would only get 1/1459 vqa answers correct and my explanations look off. In fact, when I test the training set, I only get 15/29459 correct. However, when I use your pretrained model, I get 1073/1459 correct using generate_explanations.py for the validation set.

Is there a step I'm missing going from training my own model using train.py to generating explanations using generate_explanations.py? Training more iterations (going from 30k to 50k) doesn't seem to be improving this issue.

Thanks

Hi,

I cannot seem to find the feature directory nor any softattention file. Could you please tell us where can we find this?

Thanks,
Oana

Hey Seth,

I've revisited your code again and I've been running some experiments with your dataset in pytorch! To evaluate the explanations, I've been using an adapted version of https://github.com/tylin/coco-caption. I've also used your pretrained models to generate explanations for comparisons to my results, but I notice I get numbers that are different than the reported numbers in your paper.

Using your pretrained model with --use_gt, I get 16.7 on Bleu-4 (19.8 in the paper), 51.3 for Cider (73.4 in the paper), and 39.6 for Rouge (44.0) in the paper. What evaluation code did you use to run your evaluation metrics and what do you think could be the reason from this difference?

Thanks!

Jeff

Hi,
For VQA-X, I was able to generate explanations using the pre-trained model, but I am a bit confused why it is necessary to give the --exp_file with the explanation labels. I'm referring to this command:

cd PJ-X-VQA/generate_vqa_exp
python generate_explanation.py --ques_file ../VQA-X/Questions/v2_OpenEnded_mscoco_val2014_questions.json --ann_file ../VQA-X/Annotations/v2_mscoco_val2014_annotations.json --exp_file ../VQA-X/Annotations/val_exp_anno.json --gpu 0 --out_dir ../VQA-X/results --folder ../model/ --model_path $PATH_TO_CAFFEMODEL --use_gt --save_att_map

I understand that at the moment of creating the explanations in
PJ-X-VQA/generate_vqa_exp/generate_explanation.py the labels are not being used. Could you please clarify?
Thanks,

I understand that the visual explanation (in the form of a heatmap) is given in VQA-X/visual. How do I extract the images? I am using the following numpy code:

img = np.load("11042002.npy")
shape = np.sort(img.shape)[::-1] # Reverse sorting shape so as to bring it in the form of (64x, 48x, 3)
plt.imshow(img.reshape(shape))

However, this leads to strange heatmaps like:

What am I doing wrong here?

Hi,

Could you please what are the files v2_mscoco_train2014_annotations.json and the v2_mscoco_val2014_annotations.json meant to be in Annotations/ and the v2_OpenEnded_mscoco_train2014_questions.json and v2_OpenEnded_mscoco_val2014_questions.json files from the Questions/ ?

I couldn't find these exact names in the links mentioned and I want to make sure I'm not using incorrect files if I just guess myself what they would be. Also, to be sure, should visual/val/ and visual/test be empty in case we retrain the model from scratch or use the files you in https://drive.google.com/drive/u/0/folders/1Cr9JRXDmjks_wmi-a9eIe4SWSwWKcCk7 ?

Thanks,
Oana

From where do I get the

• train_exp_anno.json

• val_exp_anno.json and

• test_exp_anno.json

Hello Seth,
I am using pyemd as you posted, but I have some doubts about the EMD calculation.
Did you use emd or emd_samples ?.
If it is emd, what distance matrix did you use?, and if it is emd_samples, how many bins ?

Thanks!

The VQA-X google drive location contains 1000 images from val and test each. Is there any way to access the rest of the data?

Hi there, in the google drive you share, there is only val & set visual explanation, could you please also share the train set? Many thanks!

Does this repo contain code to calculate the Earth Mover Distance (EMD) and Rank Correlation as given in the paper? If not, how can I go about calculating them?

Hi Seth,

I am curious about what information does visual directory contain. This would be helpful to know as I want to train this model on a completely different dataset.

Thank you!

Originally posted by @Seth-Park in #12 (comment)

How do I open to .npz file generated during feature extraction?
Thanks