moonqy / askaway-vqa_with_modifications

Developed stacked attention based VQA system with DenseNet and LSTM to study fixation on language prior for VQA 1.0 and VQA 2.0 datasets.

askaway-vqa_with_modifications

This implementation is based on the paper Show, Ask, Attend, and Answer: A Strong Baseline For Visual Question Answering in PyTorch. The reference code was taken from here. Here denset was used for extracting the image features and LSTM was used for question feature extraction. We have also experimented with GRU for question feature extraction. Lastly, we have studied the fixation of the network on language prior. For architecture details and results, please refer to the project report. The VQA 1.0 and VQA 2.0 dataset can be selected by editing the paths in config.py.

Running the model as it is

Download VQA 1.0 and 2.0 datasets from here. After setting up appropriate paths in config.py, run the following commands to obtain results on validation set:

python preprocess-images.py
python preprocess-vocab.py
python train.py

The progress can be viewed using:

python view-log.py <path to .pth log>

Using Densenet and GRU in the network

The densenet-161 architecture was being used here and the changes were made in preprocess-images.py file. The result for this change can be viewed as:

The GRU network was implemented in model.py and can be changed back to LSTM by editing the same.

Studying the fixation on language prior by turning off image pipeline

The fixation on language prior was first studied by passing a tensor of all 1s. This was done by editing data.py and image can be turned on by editing the same.

Studying the fixation on language prior by removing the image and attention pipeline

The fixation on language prior was then studied by simplifying the network and reducing it to NLP question answering task. This was done by editing model.py. The execution time was reduced 3 times here.

Visualizations

Obtained attention from AskAway for question ’Does this flask have enough wine to fill the glass?’ a) Input image b) First attention glimpse c)Second attention (finer) glimpse