Memory leak using cache_structs about libmolgrid

[dkoes]

With cache_mols enabled, your entire dataset will be loaded into memory (after processing - so just the xyz coordinates and type information). If your dataset doesn't fit in memory, you will run out. What you should see is a steady increase in memory usage until you have done one complete epoch at which point it levels off.

If your dataset isn't very large and/or you are seeing increased memory usage after performing a complete epoch, we will investigate, but otherwise it is the expected behavior.

If you aren't using vector typing, you can use the create_caches2.py script (https://github.com/gnina/scripts/blob/master/create_caches2.py) to assemble your dataset into one very efficient memory-mappable file. Memory mapping let's you benefit from in-memory data without having to worry about out-of-memory errors (and especially valuable when running multiple jobs on the same node).

[jscant]

The models I'm using are on the edge of what's possible to store in the memory that I have available, so the 23gb dataset on top of that is indeed giving me OOM problems. I suppose this can be closed, perhaps adding something to the docs warning about large datasets and the default constructor (cache_structs=True) would be useful.

Thanks for the prompt response.