ap229997 / conditional-batch-norm

Pytorch implementation of NIPS 2017 paper "Modulating early visual processing by language"

Conditional Batch Normalization

Pytorch implementation of NIPS 2017 paper "Modulating early visual processing by language" [Link]

Introduction

The authors present a novel approach to incorporate language information into extracting visual features by conditioning the Batch Normalization parameters on the language. They apply Conditional Batch Normalization (CBN) to a pre-trained ResNet and show that this significantly improves performance on visual question answering tasks.

Setup

This repository is compatible with python 2.

• Follow instructions outlined on PyTorch Homepage for installing PyTorch (Python2).
• The python packages required are nltk tqdm which can be installed using pip.
  

Data

To download the VQA dataset please use the script 'scripts/vqa_download.sh':

scripts/vqa_download.sh `pwd`/data

Process Data

Detailed instructions for processing data are provided by GuessWhatGame/vqa.

Create dictionary

To create the VQA dictionary, use the script preprocess_data/create_dico.py.

python preprocess_data/create_dictionary.py --data_dir data --year 2014 --dict_file dict.json

Create GLOVE dictionary

To create the GLOVE dictionary, download the original glove file and run the script preprocess_data/create_gloves.py.

wget http://nlp.stanford.edu/data/glove.42B.300d.zip -P data/
unzip data/glove.42B.300d.zip -d data/
python preprocess_data/create_gloves.py --data_dir data --glove_in data/glove.42B.300d.txt --glove_out data/glove_dict.pkl --year 2014

Train Model

To train the network, set the required parameters in config.json and run the script main.py.

python main.py --gpu gpu_id --data_dir data --img_dir images --config config.json --exp_dir exp --year 2014

Citation

If you find this code useful, please consider citing the original work by authors:

@inproceedings{de2017modulating,
author = {Harm de Vries and Florian Strub and J\'er\'emie Mary and Hugo Larochelle and Olivier Pietquin and Aaron C. Courville},
title = {Modulating early visual processing by language},
booktitle = {Advances in Neural Information Processing Systems 30},
year = {2017}
url = {https://arxiv.org/abs/1707.00683}
}

conditional-batch-norm's Issues

Mean and variance over all the dimensions?

These two lines in the CBN code calculates the mean and variance across all the dimensions. But shouldn't we calculate them for each channel separately ? (as per here)

MLPs for betas and gammas

In https://github.com/ap229997/Conditional-Batch-Norm/blob/master/model/cbn.py#L30,
doesn't CBN need separate MLPs to predict the deltas of betas and gammas? Otherwise, both deltas are the same.

'TypeError: an integer is required (got type NoneType)'

Conditional-Batch-Norm/model/cbn.py

Line 20 in a7d234d

self.batch_size = None

I have faced some errors around here. 'TypeError: an integer is required (got type NoneType)'

Does torch.zeros() take None type?

gammas and betas

I think this variable should be 1-dimensional, gammas, and betas should be same over the batch, right?

Conditional-Batch-Norm/model/cbn.py

Line 26 in a7d234d

self.betas = nn.Parameter(torch.zeros(self.batch_size, self.channels).cuda())

Does your code really work?

In the resnet.py, even your own test code cannot run correctly!!! The parameters between your defined functions cannot fit exactly!!!
In the cbn.py. I don't konw why you init the self.beta and self.gamma to be 2 dimensions. What's more, you calculate the mean and var of the image across all elements.That's obviouslly wrong.

What accuracy can you finally get?

Thanks for you great work, it seems very helpful!

Can you tell me what accuracy you get finally?