Minerva GPU-- free memory for a variable about minerva

[jermainewang]

Hi,

Minerva keeps track of the memory using reference counting. An array in Minerva is nothing but a pointer to a memory buffer. When the reference counts of a memory buffer drops to zero (i.e, no array is pointing to the memory buffer anymore), the memory will be released. Therefore, you do not need to explicitly free memory. For example, in your example, if the data variable is out of scope, its internal memory should be released.

So if you find the memory consumption keeps growing, you must have created too many arrays and the total memory usage is beyond the gpu memory limit, in which case there is little you could do. Please have a look at your code and calculate the total amount of memory to be used. We'd also like to have a look as well if you could narrow the problem in some short examples.

Hope this will help.
Minjie

[hjk41]

Minerva has its own memory manage system. We keep a memory pool of the
freed memory blocks, and allocates from the memory pool when we need a new
memory block.

The memory corresponding to an NArray is freed to memory pool automatically
when the NArray is not referenced anymore. Manually deleting the memory of
an NArray (as you did in free_memory) is not recommended. It will confuse
the memory management system. So both pieces of your code are doomed to
fail, although the second one happened to work. :-)

A way to limit memory usage is to limit the number of "live" NArrays in
scope. NArrays that are out of scope will be automatically GCed, and memory
will be freed. You can also call Python's GC manually if you find it a
problem.

On Fri, Aug 7, 2015 at 11:33 AM, chaocliu [email protected] wrote:

Hi

Thanks for the great tool.
I are trying to use minerva to run RNN on huge-size dataset. However, the
gpu memory increase gradually and get crashed when it is beyond the gpu
memory limit. My first question is that is there any way/function to free
memory of unused variables? I have used wait_for_all, however it cannot
solve the problem.

I try to write a memory-free function by cudaFree. The function is able to
free the memory of the Narray variable, however, the pointer to the Narray
variable still exists. When I reuse the Narray variable for assignment
operation recursively, e.g. data=owl.from_numpy(np.array([...])), I found
that it works for a new array with different size, however, it get crashed
when the size of new array is same as the old one that has been deleted.
For example:

//cannot work
data = owl.from_numpy(np.range(10000).reshape(100，100))
owl.free_memory(data) //I write the function using cudaFree by myself
data = owl.from_numpy(np.range(10000).reshape(100，100))
owl.free_memory(data) //core dump here

//can work
data = owl.from_numpy(np.arange(10000).reshape(100，100))
owl.free_memory(data)
data = owl.from_numpy(np.arange(20000).reshape(200，100))
owl.free_memory(data)

Would you please find out for me the reason? Thanks a lot

—
Reply to this email directly or view it on GitHub
#46.

HONG Chuntao
System Research Group
Microsoft Research Asia

[chaocliu]

Hi jermainewang and hjk41,

Thanks a lot for your prompt and detail reply. It is a great help for me to understand the Minerva memory management system and direct me how to revise my code. Thank you again for providing the great tool!