This is a final project about visual question answering implemented through PyTorch. The code is written based on github.
- Clone this repository with:
git clone https://github.com/yuanzhu1997/VQA --recursive
- Set the paths to your downloaded questions, answers, and abstract-scene images in
config.py
. - Pre-process images with ResNet18 weights ported from Caffe and vocabularies for questions and answers with:
python preprocess-images.py
python preprocess-vocab.py
- Train the model in
model.py
with:
python train.py model-name
Here we provide four different models.
model-name | Description |
---|---|
Base | Baseline |
LSTM-Base | Baselien w/ 1layer LSTM question encoding |
LSTM-3-Base | Baselien w/ 3layer GRU question encoding |
LSTM-bi-Base | Baselien w/ bidirection LSTM question encoding |
GRU-Base | Baselien w/ 1layer GRU question encoding |
GRU-3-Base | Baselien w/ 3layer GRU question encoding |
GRU-bi-Base | Baselien w/ bidirection GRU question encoding |
SCA | spatial-channel attention |
CSA | channel-spatial attention |
C-Base | channel attention |
S-Base | spatial only attention |
The process will alternate between training and validation process while preting out the current progress to stdout. After the training, the result will be automatically saved in the logs
directory.
The logs contain the name of the model, training statistics, contents of config.py
, model weights, evaluation information (per-question answer and accuracy), and question and answer vocabularies.
We also provide the function to visualize the spatial-wise and channel-wise attention mechansim.
python visualization.py network device image_item question_item
- torch
- torchvision
- h5py
- tqdm