Problem when using multiple GPUs about torchnet

[ksemianov]

@joeyhng: try model:zeroGradParameters() instead of grad_params:zero()

[lvdmaaten]

I may be wrong here, but it appears you're adding the network only to the second GPU, not to the first one? Can you try something like:

if nGPU > 1 then
   local singlemodel = model
   local model = nn.DataParallelTable(1, true, false)
   for i = 1, nGPU do
      cutorch.setDevice(i)
      model:add(singlemodel:clone():cuda(), i)
   end
   cutorch.setDevice(1)
end

[joeyhng]

I think the way to use multiple GPU should be correct. I'm just following fb.resnet.torch
https://github.com/facebook/fb.resnet.torch/blob/master/models/init.lua#L81

[lvdmaaten]

I tried running the code example in your post, but I can not reproduce the issue you are seeing. Output:

Number of parameters:   11184650
iteration 1: loss=2.352886
iteration 2: loss=2.275253
iteration 3: loss=2.204911
iteration 4: loss=2.140517
iteration 5: loss=2.081086

[lvdmaaten]

Closing this for lack of a repro or response.

[foelin]

Have you solved the problem? I also get nan when training with multiple GPU.
Thanks

[joeyhng]

No, I haven't solved the problem. I'm still interested in solving it as well.

…

On Fri, Dec 9, 2016, 4:44 AM Zhenhua Wang ***@***.***> wrote: Have you solved the problem? I also get nan when training with multiple GPU. Thanks — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#56 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABOuSdRa7JZMrYPuSE4AV72SdEZ8GEAoks5rGGwdgaJpZM4J8Ndi> .

[joeyhng]

That seems to be the reason. It's fixed for me now. Thanks very much!!

[foelin]

it also works for me! Thanks!