mateuszmalinowski / visual_turing_test-tutorial

Tutorial for Visual Turing Test (visual question answering, image question answering).

Tutorial

To run this tutorial you need to run jupyter.

If you run jupyter remotely, you can use jupyter notebook --ip=0.0.0.0

Main file: visual_turing_test.ipynb The reader is however encouraged to download the notebook together with the associated files and go through the tutorial on his own.

The tutorial should be run on a Linux machine. Please also make sure that all Installation requirements are fullfiled and you have similar versions of Theano and Keras (see 'Tested on').

Please contact [email protected] if you encounter any problems.

Kraino - Keras-based RNN for Visual Turing Test

Keras implementation of the 'Ask Your Neurons'.

• Free software: MIT license
• If you use this library, please cite our "Ask Your Neurons" paper [1]
• Note that we use a simplified version of Kraino for the purpose of the Tutorial

Installation

Requirements:

• Theano
• Keras (fchollet)
• toolz
• h5py
• Bokeh (0.10.0)
• nltk (required by WUPS metrics)
• pydot
• spacy

Additional:

• VQA (VT-vision-lab/VQA) for Visual Question Answering
• vqaEvaluation for the evaluation metrics
• vqaTools for the dataset providers
• both should be placed in the kraino/utils folder

Folders structure

data/

daquar/

vqa/

...

kraino/

local/

    logs/

    weights/

    model-*.pkl

kraino/

    __init__.py

    core/

    utils/

data

• store all datasets

kraino

• source code and local ouput
• local
  • stores logs (e.g. predictions) in the 'logs' folder
  • stores weights of different models in the 'weights' folder
  • stores model topologies as '.pkl' files
• kraino
  • stores the models in the 'core' folder
  • stores functions (dataset providers or callbacks) in the 'utils' folder

Eras

It counts a computational cycle in eras (not epochs). Every era ends when "MAX EPOCH" is reached, then the training proceeds to the next era. Before and after each era the (callback) actions are executed.

Warning

The framework is under the continous development, and hence it is not warranted that API won't change in the future. To avoid adaptations to new API, you can clone from a specific commit hash.

Tested on

• Python 2.7.3
• Theano:0.8.0.dev0.dev-63990436c98f107cf120f3578021a5d259ecf352
• Keras:b587aeee1c1be3633a56b945af3e7c2c303369ca

Bibliography

@article{malinowski2016ask,

   title={Ask Your Neurons: A Deep Learning Approach to Visual Question Answering},
   
   author={Malinowski, Mateusz and Rohrbach, Marcus and Fritz, Mario},

   journal={arXiv preprint arXiv:1605.02697},
   
   year={2016}
   
}

@inproceedings{malinowski2015ask,

    title={Ask your neurons: A neural-based approach to answering questions about images},

    author={Malinowski, Mateusz and Rohrbach, Marcus and Fritz, Mario},

    booktitle={Proceedings of the IEEE International Conference on Computer Vision},

    pages={1--9},

    year={2015}

}

@inproceedings{malinowski2014multi,

  title={A multi-world approach to question answering about real-world scenes based on uncertain input},
  
  author={Malinowski, Mateusz and Fritz, Mario},
  
  booktitle={Advances in Neural Information Processing Systems},
  
  pages={1682--1690},
  
  year={2014}
  
}

@article{malinowski2016tutorial,

  title={Tutorial on Answering Questions about Images with Deep Learning},
  
  author={Malinowski, Mateusz and Fritz, Mario},
  
  journal={arXiv preprint arXiv:1610.01076},
  
  year={2016}
  
}

visual_turing_test-tutorial's Issues

Error in the code!

the word2index of the answers in the notebook is wrong because wordcount is not right.
Look at the 2 keys "alarm" and "alarm,"

i think the solution is to fix this function
frequencies(' '.join(train_raw_y).split(' '))

we must split the answers by the ' ' and ',' NOT just ' ' alone
Am i right?

error when running the model.compile function

when i run the following block of code from the tutorial i see an error.

model_config = Config(
    textual_embedding_dim=500,
    input_dim=len(word2index_x.keys()),
    output_dim=len(word2index_y.keys()))

model = BlindBOW(model_config)
model.create()

model.compile(
    loss='categorical_crossentropy', 
    optimizer='adam')
text_bow_model = model

---

TypeError Traceback (most recent call last)
in ()
9 model.compile(
10 loss='categorical_crossentropy',
---> 11 optimizer='adam')
12 text_bow_model = model

/home/bassel/anaconda/lib/python2.7/site-packages/keras/models.pyc in compile(self, optimizer, loss, class_mode, sample_weight_mode, **kwargs)
505 self.X_test = self.get_input(train=False)
506
--> 507 self.y_train = self.get_output(train=True)
508 self.y_test = self.get_output(train=False)
509

/home/bassel/anaconda/lib/python2.7/site-packages/keras/layers/containers.pyc in get_output(self, train)
128
129 def get_output(self, train=False):
--> 130 return self.layers[-1].get_output(train)
131
132 def set_input(self):

/home/bassel/anaconda/lib/python2.7/site-packages/keras/layers/core.pyc in get_output(self, train)
735
736 def get_output(self, train=False):
--> 737 X = self.get_input(train)
738 return self.activation(X)
739

/home/bassel/anaconda/lib/python2.7/site-packages/keras/layers/core.pyc in get_input(self, train)
239 if previous_layer_id in self.layer_cache:
240 return self.layer_cache[previous_layer_id]
--> 241 previous_output = self.previous.get_output(train=train)
242 if self.layer_cache is not None and self.cache_enabled:
243 previous_layer_id = '%s_%s' % (id(self.previous), train)

/home/bassel/anaconda/lib/python2.7/site-packages/keras/layers/core.pyc in get_output(self, train)
1028
1029 def get_output(self, train=False):
-> 1030 X = self.get_input(train)
1031 output = self.activation(K.dot(X, self.W) + self.b)
1032 return output

/home/bassel/anaconda/lib/python2.7/site-packages/keras/layers/core.pyc in get_input(self, train)
239 if previous_layer_id in self.layer_cache:
240 return self.layer_cache[previous_layer_id]
--> 241 previous_output = self.previous.get_output(train=train)
242 if self.layer_cache is not None and self.cache_enabled:
243 previous_layer_id = '%s_%s' % (id(self.previous), train)

/home/bassel/anaconda/lib/python2.7/site-packages/keras/layers/core.pyc in get_output(self, train)
701
702 def get_output(self, train=False):
--> 703 X = self.get_input(train)
704 if self.p > 0.:
705 if train:

/home/bassel/anaconda/lib/python2.7/site-packages/keras/layers/core.pyc in get_input(self, train)
239 if previous_layer_id in self.layer_cache:
240 return self.layer_cache[previous_layer_id]
--> 241 previous_output = self.previous.get_output(train=train)
242 if self.layer_cache is not None and self.cache_enabled:
243 previous_layer_id = '%s_%s' % (id(self.previous), train)

/home/bassel/anaconda/lib/python2.7/site-packages/keras/layers/core.pyc in get_output(self, train)
229
230 def get_output(self, train=False):
--> 231 return self.get_input(train)
232
233 def get_input(self, train=False):

/home/bassel/anaconda/lib/python2.7/site-packages/keras/layers/core.pyc in get_input(self, train)
239 if previous_layer_id in self.layer_cache:
240 return self.layer_cache[previous_layer_id]
--> 241 previous_output = self.previous.get_output(train=train)
242 if self.layer_cache is not None and self.cache_enabled:
243 previous_layer_id = '%s_%s' % (id(self.previous), train)

/home/bassel/Desktop/Graduation_project_2/visual_turing_test-tutorial-master/kraino/core/keras_extensions.py in get_output(self, train)
111 if hasattr(self, 'previous'):
112 return func(self.previous.get_output(train),
--> 113 self.previous.get_output_mask(train))
114 else:
115 return func(self.input, self.get_output_mask(train))

/home/bassel/Desktop/Graduation_project_2/visual_turing_test-tutorial-master/kraino/core/keras_extensions.py in time_distributed_masked_ave(x, m)
51 """
52 tmp = K.sum(x, axis=1)
---> 53 nonzeros = K.sum(m, axis=-1)
54 return tmp / K.expand_dims(K.cast(nonzeros, tmp.dtype))
55

/home/bassel/anaconda/lib/python2.7/site-packages/keras/backend/tensorflow_backend.pyc in sum(x, axis, keepdims)
149 '''
150 axis = normalize_axis(axis, ndim(x))
--> 151 return tf.reduce_sum(x, reduction_indices=axis, keep_dims=keepdims)
152
153

/home/bassel/anaconda/lib/python2.7/site-packages/tensorflow/python/ops/math_ops.pyc in reduce_sum(input_tensor, axis, keep_dims, name, reduction_indices)
1173 _ReductionDims(input_tensor, axis, reduction_indices),
1174 keep_dims,
-> 1175 name=name)
1176
1177

/home/bassel/anaconda/lib/python2.7/site-packages/tensorflow/python/ops/gen_math_ops.pyc in _sum(input, reduction_indices, keep_dims, name)
2785 result = _op_def_lib.apply_op("Sum", input=input,
2786 reduction_indices=reduction_indices,
-> 2787 keep_dims=keep_dims, name=name)
2788 return result
2789

/home/bassel/anaconda/lib/python2.7/site-packages/tensorflow/python/framework/op_def_library.pyc in apply_op(self, op_type_name, name, **keywords)
580 for base_type in base_types:
581 _SatisfiesTypeConstraint(base_type,
--> 582 _Attr(op_def, input_arg.type_attr))
583 attrs[input_arg.type_attr] = attr_value
584 inferred_from[input_arg.type_attr] = input_name

/home/bassel/anaconda/lib/python2.7/site-packages/tensorflow/python/framework/op_def_library.pyc in _SatisfiesTypeConstraint(dtype, attr_def)
58 "DataType %s for attr '%s' not in list of allowed values: %s" %
59 (dtypes.as_dtype(dtype).name, attr_def.name,
---> 60 ", ".join(dtypes.as_dtype(x).name for x in allowed_list)))
61
62

TypeError: DataType bool for attr 'T' not in list of allowed values: float32, float64, int64, int32, uint8, uint16, int16, int8, complex64, complex128, qint8, quint8, qint32, float16

ImportError: cannot import name TimeDistributedMerge

Hi,

I was trying to run your excellent work. Meanwhile I installed all the dependencies but I got an error when I run the code snippet of 'class BlindBOW'. The code and respective output is given. Your suggestion is highly appreciated.
`#== Model definition

from keras.layers.core import Activation
from keras.layers.core import Dense
from keras.layers.core import Dropout
from keras.layers.core import TimeDistributedMerge
from keras.layers.embeddings import Embedding

from kraino.core.model_zoo import AbstractSequentialModel
from kraino.core.model_zoo import AbstractSingleAnswer
from kraino.core.model_zoo import AbstractSequentialMultiplewordAnswer
from kraino.core.model_zoo import Config
from kraino.core.keras_extensions import DropMask
from kraino.core.keras_extensions import LambdaWithMask
from kraino.core.keras_extensions import time_distributed_masked_ave

class BlindBOW(AbstractSequentialModel, AbstractSingleAnswer):
"""
BOW Language only model that produces single word answers.
"""
def create(self):
self.add(Embedding(
self._config.input_dim,
self._config.textual_embedding_dim,
mask_zero=True))
self.add(LambdaWithMask(time_distributed_masked_ave, output_shape=[self.output_shape[2]]))
self.add(DropMask())
self.add(Dropout(0.5))
self.add(Dense(self._config.output_dim))
self.add(Activation('softmax'))`

Output Error:
`ImportError Traceback (most recent call last)
in ()
5 from keras.layers.core import Dense
6 from keras.layers.core import Dropout
----> 7 from keras.layers.core import TimeDistributedMerge
8 from keras.layers.embeddings import Embedding
9

ImportError: cannot import name TimeDistributedMerge
`