Memory leaks in AdamTrainer with RegressionLayer about convnetsharp

[bjornbergq]

No It's the list of Volumes for gmax and smax that is not disposed(?) correctly. I don't see a reason why it should dispose them but during training of my model I found that the GPU-class has ALOT of memory leaks. I fixed some of them, but this one is a mystery to me.

It creates a list of volumes, references the layers gradients and updates it, never destroying the list. BUT somewhere along the line it uses more and more GPU Memory. I think it has to do with how the GPU-class adopts the volume, maybe it gets a new pointer in memory after all(?).

[bjornbergq]

OH, after a quick look again I think i found it. It restores the Storage on the device, making duplicates of the volume, instead of having a pointer to the same location. So do it even update the same values then?

[bjornbergq]

Correction!

The other stuff was correct. But the adam trainer clones preivous volumes, hence creates more memory. Perahps the .Clone part should not be there?

[cbovar]

Do you have a small sample that reproduces the leak ? I will look if that Clone is necessary.

I have merged a pull request just now that solves some GPU memory problem => Could you test with latest code?

Also, in Debug, you will always see lots of transfer between host and device because of here and here

[cbovar]

I haven't found the leak yet. However there is a lot of volume instantiations in the trainers. I will introduce a pool of Volumes in VolumeBuilder to recycle them rather than disposing them and allocating them again.