abhshkdz / neural-vqa Goto Github PK

DaviddeMacBook-Pro:neural-vqa-master lihong$ th predict.lua -checkpoint_file vqa_epoch15.00_0.4576.t7 -input_image_path data/train2014/COCO_train2014_000000405541.jpg -question 'What is the cat on?'
/Users/lihong/torch/install/bin/luajit: ./utils/DataLoader.lua:35: attempt to index local 'train_questions_attr' (a nil value)
stack traceback:
./utils/DataLoader.lua:35: in function 'create'
predict.lua:59: in main chunk
[C]: in function 'dofile'
...hong/torch/install/lib/luarocks/rocks/trepl/scm-1/bin/th:145: in main chunk
[C]: at 0x010ad1fbd0

rzai@rzai00:~/prj/neural-vqa$ tree
.
├── data
│   ├── download_data.sh
│   ├── mscoco_train2014_annotations.json
│   ├── mscoco_val2014_annotations.json
│   ├── MultipleChoice_mscoco_train2014_questions.json
│   ├── MultipleChoice_mscoco_val2014_questions.json
│   ├── OpenEnded_mscoco_train2014_questions.json
│   ├── OpenEnded_mscoco_val2014_questions.json
│   ├── train2014
│   └── val2014
├── evaluate.lua
├── extract_fc7.lua
├── lstm.lua
├── models
│   ├── download_models.sh
│   ├── VGG_ILSVRC_19_layers.caffemodel -> /media/rzai/ai_data/VGG_ILSVRC_19_layers.caffemodel
│   └── VGG_ILSVRC_19_layers_deploy.prototxt
├── predict.lua
├── README.md
├── train.lua
└── utils
├── DataLoader.lua
├── JSON.lua
└── misc.lua

3 directories, 21 files
rzai@rzai00:~/prj/neural-vqa$

Based on your code, I think it is very natural and easy to modify it to do the task of image caption. Here is the version I implemented, but currently the implementation is flawed because the performance is bad. can U take some time to review it and give me some suggestions, here is the code link:
https://github.com/eriche2016/Image-caption-version2

rzai@rzai00:~/prj/neural-vqa$ CUDA_VISIBLE_DEVICES=0 th train.lua -gpuid 0
{
beta : 0.999
train_fc7_file : "data/train_fc7.t7"
embedding_size : 512
val_fc7_file : "data/val_fc7.t7"
dropout : 0.5
seed : 981723
data_dir : "data"
learning_rate_decay : 0.95
epsilon : 1e-08
learning_rate_decay_after : 15
max_epochs : 50
checkpoint_dir : "checkpoints"
rnn_size : 512
alpha : 0.8
num_layers : 2
init_from : ""
savefile : "vqa"
batch_size : 2
val_fc7_image_id_file : "data/val_fc7_image_id.t7"
train_fc7_image_id_file : "data/train_fc7_image_id.t7"
save_every : 1000
learning_rate : 0.0004
gpuid : 0
}
not found: THNN_CudaHalfELU_updateOutput/home/rzai/torch/install/share/lua/5.1/nn/THNN.lua:108: NYI: call arg type
not found: THNN_CudaHalfELU_updateGradInput/home/rzai/torch/install/share/lua/5.1/nn/THNN.lua:108: NYI: call arg type
not found: THNN_CudaHalfHardTanh_updateOutput/home/rzai/torch/install/share/lua/5.1/nn/THNN.lua:108: NYI: call arg type
not found: THNN_CudaHalfHardTanh_updateGradInput/home/rzai/torch/install/share/lua/5.1/nn/THNN.lua:108: NYI: call arg type
not found: THNN_CudaHalfLeakyReLU_updateOutput/home/rzai/torch/install/share/lua/5.1/nn/THNN.lua:108: NYI: call arg type
not found: THNN_CudaHalfLeakyReLU_updateGradInput/home/rzai/torch/install/share/lua/5.1/nn/THNN.lua:108: NYI: call arg type
not found: THNN_CudaHalfLookupTable_accGradParameters/home/rzai/torch/install/share/lua/5.1/nn/THNN.lua:108: NYI: call arg type
not found: THNN_CudaHalfLookupTable_renorm/home/rzai/torch/install/share/lua/5.1/nn/THNN.lua:108: NYI: call arg type
not found: THNN_CudaHalfMarginCriterion_updateOutput/home/rzai/torch/install/share/lua/5.1/nn/THNN.lua:108: NYI: call arg type
not found: THNN_CudaHalfMarginCriterion_updateGradInput/home/rzai/torch/install/share/lua/5.1/nn/THNN.lua:108: NYI: call arg type
not found: THNN_CudaHalfMultiMarginCriterion_updateOutput/home/rzai/torch/install/share/lua/5.1/nn/THNN.lua:108: NYI: call arg type
not found: THNN_CudaHalfMultiMarginCriterion_updateGradInput/home/rzai/torch/install/share/lua/5.1/nn/THNN.lua:108: NYI: call arg type
not found: THNN_CudaHalfPReLU_accGradParameters/home/rzai/torch/install/share/lua/5.1/nn/THNN.lua:108: NYI: call arg type
not found: THNN_CudaHalfSpatialConvolutionLocal_accGradParameters/home/rzai/torch/install/share/lua/5.1/nn/THNN.lua:108: NYI: call arg type
not found: THNN_CudaHalfSpatialConvolutionMM_accGradParameters/home/rzai/torch/install/share/lua/5.1/nn/THNN.lua:108: NYI: call arg type
not found: THNN_CudaHalfSpatialCrossMapLRN_updateOutput/home/rzai/torch/install/share/lua/5.1/nn/THNN.lua:108: NYI: call arg type
not found: THNN_CudaHalfSpatialCrossMapLRN_updateGradInput/home/rzai/torch/install/share/lua/5.1/nn/THNN.lua:108: NYI: call arg type
not found: THNN_CudaHalfSpatialDilatedConvolution_accGradParameters/home/rzai/torch/install/share/lua/5.1/nn/THNN.lua:108: NYI: call arg type
not found: THNN_CudaHalfSpatialFullConvolution_accGradParameters/home/rzai/torch/install/share/lua/5.1/nn/THNN.lua:108: NYI: call arg type
not found: THNN_CudaHalfSoftPlus_updateOutput/home/rzai/torch/install/share/lua/5.1/nn/THNN.lua:108: NYI: call arg type
not found: THNN_CudaHalfSoftPlus_updateGradInput/home/rzai/torch/install/share/lua/5.1/nn/THNN.lua:108: NYI: call arg type
not found: THNN_CudaHalfSoftShrink_updateOutput/home/rzai/torch/install/share/lua/5.1/nn/THNN.lua:108: NYI: call arg type
not found: THNN_CudaHalfSoftShrink_updateGradInput/home/rzai/torch/install/share/lua/5.1/nn/THNN.lua:108: NYI: call arg type
not found: THNN_CudaHalfSqrt_updateOutput/home/rzai/torch/install/share/lua/5.1/nn/THNN.lua:108: NYI: call arg type
not found: THNN_CudaHalfTemporalConvolution_accGradParameters/home/rzai/torch/install/share/lua/5.1/nn/THNN.lua:108: NYI: call arg type
not found: THNN_CudaHalfThreshold_updateOutput/home/rzai/torch/install/share/lua/5.1/nn/THNN.lua:108: NYI: call arg type
not found: THNN_CudaHalfThreshold_updateGradInput/home/rzai/torch/install/share/lua/5.1/nn/THNN.lua:108: NYI: call arg type
not found: THNN_CudaHalfVolumetricConvolution_accGradParameters/home/rzai/torch/install/share/lua/5.1/nn/THNN.lua:108: NYI: call arg type
not found: THNN_CudaHalfVolumetricDilatedConvolution_accGradParameters/home/rzai/torch/install/share/lua/5.1/nn/THNN.lua:108: NYI: call arg type
not found: THNN_CudaHalfVolumetricFullConvolution_accGradParameters/home/rzai/torch/install/share/lua/5.1/nn/THNN.lua:108: NYI: call arg type
using CUDA on GPU 0...
Loading data files...
Loading train fc7 features from data/train_fc7.t7

Great work and very informative accompanying readme file!

However, I think there is an issue in extracting image features using extract_fc7.lua on CPU.

extract_fc7.lua:123: attempt to index global 'cutorch' (a nil value)

I think this issue can be resolved by substituting line 123 cutorch.synchronize() by
if opt.gpuid >= 0 then cutorch.synchronize() end

Thanks!

I am trying to get the results through pretrained cpu checkpoint. My command is

th predict.lua -checkpoint_file checkpoints/vqa_epoch23.26_0.4610_cpu.t7 -input_image_path data/train2014/COCO_train2014_000000405541.jpg -question 'What is the cat on?' -gpuid -1

Error given is
Loading data files...
/home/aekansh/torch/install/bin/lua: ...e/aekansh/torch/install/share/lua/5.1/torch/File.lua:259: read error: read 0 blocks instead of 1 at /home/aekansh/torch/pkg/torch/lib/TH/THDiskFile.c:349
stack traceback:
[C]: in function 'readInt'
...e/aekansh/torch/install/share/lua/5.1/torch/File.lua:259: in function 'readObject'
...e/aekansh/torch/install/share/lua/5.1/torch/File.lua:368: in function 'readObject'
...e/aekansh/torch/install/share/lua/5.1/torch/File.lua:369: in function 'readObject'
...e/aekansh/torch/install/share/lua/5.1/torch/File.lua:369: in function 'readObject'
...e/aekansh/torch/install/share/lua/5.1/torch/File.lua:409: in function 'load'
./utils/DataLoader.lua:47: in function 'create'
predict.lua:59: in main chunk
[C]: in function 'dofile'
.../torch/install/lib/luarocks/rocks/trepl/scm-1/bin/th:150: in main chunk
[C]: ?

Even if I donot use -gpuid parameter it gives the same error

while extracting the image features using th extract_fc7.lua -split train
bad argument 3 to 'narrow' out of range at (path_to_torch/pkg/torch/lib/TH/generic/THTensor.c:351)
above error is thrown

cannot open <data/MultipleChoice_mscoco_train2014_questions.json> in mode r at /home/hadjer/torch/pkg/torch/lib/TH/THDiskFile.c:673
stack traceback:
[C]: ?
[C]: in function 'DiskFile'
./utils/DataLoader.lua:234: in function 'json_to_tensor'
./utils/DataLoader.lua:43: in function 'create'
extract_fc7.lua:47: in main chunk
[C]: in function 'dofile'
.../torch/install/lib/luarocks/rocks/trepl/scm-1/bin/th:150: in main chunk
Hello,
I had this problem when I installed lua5.1
could someone help to solve this issue plz

Hello,
how can I change this part of code to forward the image fisrt
also could u plz provide 2vis-lstm code ?
for t = 1, loader.q_max_length do
lst = lstm_clones[t]:forward{qf:select(2,t), unpack(rnn_state[t-1])}
-- at every time step, set the rnn state (h_t, c_t) to be passed as input in next time step
rnn_state[t] = {}
for i = 1, #init_state do table.insert(rnn_state[t], lst[i]) end
end

-- after completing the unrolled LSTM forward pass with question features, forward the image features
lst = lstm_clones[loader.q_max_length + 1]:forward{imf, unpack(rnn_state[loader.q_max_length])}

Error detail analysis:

$th train.lua -gpuid 0

using CUDA on GPU 0...
Loading data files...
Loading train fc7 features from data/train_fc7.t7
Loading val fc7 features from data/val_fc7.t7
Parameters: 6813673
Batches: 1076
Max iterations: 53800
/home/cse/torch/install/bin/luajit:
/home/cse/torch/install/share/lua/5.1/nn/Container.lua:67:
In 1 module of nn.Sequential:
/home/cse/torch/install/share/lua/5.1/nn/LookupTable.lua:75: bad argument
#3 to 'index' (Tensor | LongTensor expected, got torch.CudaLongTensor)

stack traceback:
[C]: in function 'index'
/home/cse/torch/install/share/lua/5.1/nn/LookupTable.lua:75: in
function
[C]: in function 'xpcall'
/home/cse/torch/install/share/lua/5.1/nn/Container.lua:63: in function
'rethrowErrors'
/home/cse/torch/install/share/lua/5.1/nn/Sequential.lua:44: in function
'forward'
train.lua:375: in function 'opfunc'
/home/cse/torch/install/share/lua/5.1/optim/adam.lua:37: in function
'adam'
train.lua:476: in main chunk
[C]: in function 'dofile'
.../cse/torch/install/lib/luarocks/rocks/trepl/scm-1/bin/th:145: in
main chunk
[C]: at 0x00405970

WARNING: If you see a stack trace below, it doesn't point to the place
where this error occurred. Please use only the one above.
stack traceback:
[C]: in function 'error'
/home/cse/torch/install/share/lua/5.1/nn/Container.lua:67: in function
'rethrowErrors'
/home/cse/torch/install/share/lua/5.1/nn/Sequential.lua:44: in function
'forward'
train.lua:375: in function 'opfunc'
/home/cse/torch/install/share/lua/5.1/optim/adam.lua:37: in function
'adam'
train.lua:476: in main chunk
[C]: in function 'dofile'
.../cse/torch/install/lib/luarocks/rocks/trepl/scm-1/bin/th:145: in
main chunk
[C]: at 0x00405970

Reffered : I have searched this problem and found following link :

https://groups.google.com/forum/#!topic/torch7/6igs4XdoSW4
jcjohnson/neural-style#210
jcjohnson/torch-rnn#56
https://groups.google.com/forum/#!topic/torch7/uEeBWwkyDPU

Solution : After ,I referred those link, i have re-installed following packages

luarocks install nn
luarocks install cunn
luarocks install cunn 1.0-0
luarocks install torch
luarocks install cutorch
luarocks install cutorch 1.0-0

But, I got same error. Please let me know how do i solve this issue.

Hi Abhishek,

Interesting project. Have you evaluated your model on the VQA test set? If yes, what accuracy did you obtain?

Thanks,
Avi Singh

Hello,

I didn't encounter any problems while running the example code, but the answers always come out as yes or no.

th predict.lua -checkpoint_file checkpoints/vqa_epoch23.26_0.4610.t7 -input_image_path data/train2014/COCO_train2014_000000405541.jpg -question 'What is the cat on?'

using CUDA on GPU 0...
Loading data files...
loading checkpoint from checkpoints/vqa_epoch23.26_0.4610.t7
Vocab size of checkpoint and data are different.
[libprotobuf WARNING google/protobuf/io/coded_stream.cc:537] Reading dangerously large protocol message. If the message turns out to be larger than 1073741824 bytes, parsing will be halted for security reasons. To increase the limit (or to disable these warnings), see CodedInputStream::SetTotalBytesLimit() in google/protobuf/io/coded_stream.h.
[libprotobuf WARNING google/protobuf/io/coded_stream.cc:78] The total number of bytes read was 574671192
Successfully loaded models/VGG_ILSVRC_19_layers.caffemodel
conv1_1: 64 3 3 3
conv1_2: 64 64 3 3
conv2_1: 128 64 3 3
conv2_2: 128 128 3 3
conv3_1: 256 128 3 3
conv3_2: 256 256 3 3
conv3_3: 256 256 3 3
conv3_4: 256 256 3 3
conv4_1: 512 256 3 3
conv4_2: 512 512 3 3
conv4_3: 512 512 3 3
conv4_4: 512 512 3 3
conv5_1: 512 512 3 3
conv5_2: 512 512 3 3
conv5_3: 512 512 3 3
conv5_4: 512 512 3 3
fc6: 1 1 25088 4096
fc7: 1 1 4096 4096
fc8: 1 1 4096 1000
yes

Do you have an idea how to solve the problem?

I am running this command "th extract_fc7.lua -split train" is taking very long time about 4days, and also I have 16gb ram and it is growing the size fairly quickly and is getting to out of memory within 1day.

I have a i5, 16gb RAM machine, can I run the training completely? if not what kind of machine I need.

Please help.

envy@ub1404:/os_pri/github/neural-vqa$ th extract_fc7.lua -split train
questions_vocab.t7, answers_vocab.t7 or data.t7 files do not exist. Running preprocessing...
one-time setup: preprocessing...
creating vocabulary mapping...
/home/envy/torch/install/bin/luajit: not enough memory
envy@ub1404:/os_pri/github/neural-vqa$
envy@ub1404:/os_pri/github/neural-vqa$
envy@ub1404:/os_pri/github/neural-vqa$ th extract_fc7.lua -split val
questions_vocab.t7, answers_vocab.t7 or data.t7 files do not exist. Running preprocessing...
one-time setup: preprocessing...
creating vocabulary mapping...
/home/envy/torch/install/bin/luajit: not enough memory
envy@ub1404:~/os_pri/github/neural-vqa$

My Machine:
i7 6700HQ
16GB
GTX 950M

Using the provided checkpoint with predict.

loading checkpoint from vqa_epoch23.49_0.5031.t7    
/root/torch/install/bin/luajit: /root/torch/install/share/lua/5.1/torch/File.lua:294: unknown object
stack traceback:
    [C]: in function 'error'
    /root/torch/install/share/lua/5.1/torch/File.lua:294: in function 'readObject'
    /root/torch/install/share/lua/5.1/torch/File.lua:240: in function 'readObject'
    /root/torch/install/share/lua/5.1/torch/File.lua:288: in function 'readObject'
    /root/torch/install/share/lua/5.1/torch/File.lua:272: in function 'readObject'
    /root/torch/install/share/lua/5.1/torch/File.lua:288: in function 'readObject'
    /root/torch/install/share/lua/5.1/torch/File.lua:288: in function 'readObject'
    /root/torch/install/share/lua/5.1/torch/File.lua:288: in function 'readObject'
    /root/torch/install/share/lua/5.1/torch/File.lua:288: in function 'readObject'
    /root/torch/install/share/lua/5.1/torch/File.lua:288: in function 'readObject'
    /root/torch/install/share/lua/5.1/torch/File.lua:288: in function 'readObject'
    ...
    /root/torch/install/share/lua/5.1/torch/File.lua:288: in function 'readObject'
    /root/torch/install/share/lua/5.1/nngraph/gmodule.lua:402: in function 'read'
    /root/torch/install/share/lua/5.1/torch/File.lua:270: in function 'readObject'
    /root/torch/install/share/lua/5.1/torch/File.lua:288: in function 'readObject'
    /root/torch/install/share/lua/5.1/torch/File.lua:288: in function 'readObject'
    /root/torch/install/share/lua/5.1/torch/File.lua:319: in function 'load'
    predict.lua:63: in main chunk
    [C]: in function 'dofile'
    /root/torch/install/lib/luarocks/rocks/trepl/scm-1/bin/th:145: in main chunk
    [C]: at 0x00406670

Also it says
To reproduce results shown on this page or try your own image-question pairs, download the following and run predict.lua with the appropriate paths.

What are the appropriate paths for answers_vocab.t7, questions_vocab.t7 and data.t7?

Btw training also doesn't work, i.e. the loss is nan:

Parameters: 6867121 
Batches: 3363   
Max iterations: 168150  
epoch 0.0029735355337496 loss nan
epoch 0.0059470710674993 loss nan   
epoch 0.0089206066012489 loss nan   
epoch 0.011894142134999 loss nan    
epoch 0.014867677668748 loss nan