Bug in DataHandler about deepnet

[nitishsrivastava]

There is no bug. The 10th line in your code is not necessary as long as the main memory is more than the gpu memory (which is usually the case). If you remove that line, the code will load the next files into the cache only if there are more files to be loaded (self._maxpos < self.parent._maxpos). If the whole dataset fits on the GPU (self._maxpos == self.parent._maxpos), it won't do anything avoiding the overhead of moving the same data repeatedly from the main memory.

[phvu]

Thank you for your clarification, I understand the code better now.
Unfortunately in my case, I am using Debian 32bits with some freaking limits of memory that Python can take. Therefore I can only have 0.5 GB of RAM for storing data (I put "0.5G" in the data proto file), and that makes the Python process use 2GB of RAM, which is the limit. While I have > 2GB memory on GPU, this limit is a pain. And yes, unfortunately, my data can not fit on GPU or CPU memory...

Anyway in this case, do you think it is safe to workaround like that?

[nitishsrivastava]

Yes, that would be a safe work around.