VQA_LSTM_CNN going out of memory on Titan X with 12 GB RAM about vqa_lstm_cnn

[jiasenlu]

Hi, That's wired, with the default batch size (500) it takes about 9 GB GPU RAM to run, I've also tested on Titan X. Could you try again with the smaller batch size (such as 128) and make sure no other program running on that GPU?

[jnhwkim]

@kumarabhinavgupta In my case, it works fine. To make sure your situation, update cunn and cutorch using luarocks install cunn and luarocks install cutorch.

[kumarabhinavgupta]

Thanks for replying.

We updated cunn and cutorch as suggested by @jnhwkim and tried with batch size of 128 (and even 1) as suggested by @jiasenlu . But still we are getting the "Out of Memory" error, but this time after 900 iterations.

The same Titan X is training other networks which require 8-9GB of RAMs, without a problem.

What are the likely errors which we might look for ?

[jiasenlu]

I think maybe it's safer to add collectgarbage() inside the training function. Could you try adding the following in the training function?

if i%50 == 0 then
collectgarbage()
end

or you can re-download the train.lua and try again.

[kumarabhinavgupta]

Thanks a lot. The solution is training now.

We had to make 2 more changes.

• It should be "iter" instead of i
• Remove "end" at line no. 310

[jnhwkim]

I've faced the same situation in different machine. In my case, I have to collect garbages at every 5 iterations. FYI.