This implementation is based on the paper Show, Ask, Attend, and Answer: A Strong Baseline For Visual Question Answering in PyTorch. The reference code was taken from here. Here denset was used for extracting the image features and LSTM was used for question feature extraction. We have also experimented with GRU for question feature extraction. Lastly, we have studied the fixation of the network on language prior. For architecture details and results, please refer to the project report. The VQA 1.0 and VQA 2.0 dataset can be selected by editing the paths in config.py.
Download VQA 1.0 and 2.0 datasets from here. After setting up appropriate paths in config.py, run the following commands to obtain results on validation set:
python preprocess-images.py
python preprocess-vocab.py
python train.py
The progress can be viewed using:
python view-log.py <path to .pth log>
The densenet-161 architecture was being used here and the changes were made in preprocess-images.py file. The result for this change can be viewed as:
The GRU network was implemented in model.py and can be changed back to LSTM by editing the same.
The fixation on language prior was first studied by passing a tensor of all 1s. This was done by editing data.py and image can be turned on by editing the same.
The fixation on language prior was then studied by simplifying the network and reducing it to NLP question answering task. This was done by editing model.py. The execution time was reduced 3 times here.
Obtained attention from AskAway for question βDoes this flask have enough wine to fill the glass?β a) Input image b) First attention glimpse c)Second attention (finer) glimpse