out of memory for fine tuning about fewshot_detection

[quanvuong]

you can add it after loss.backward()

[quanvuong]

adding del loss and torch.cuda.empty_cache() solve this problem

[quanvuong]

Actually, using empty_cached() leads to really slow GPU operations (60 hours for the fine tuning step). Is there another work around?

If I simply do del loss without emptying the cache, the out of memory error still happens.

[shenglih]

torch.cuda.empty_cache()

hi @quanvuong would you mind elaborating where to add these, much appreciated!

[thsunkid]

Hi @quanvuong,

Had you solved this problem ?
I got one similar when evaluating the baseline model, which caused CUDA error: out of memory due to accumulate the data from each iter.
I used torch 0.4.1 version. Already try to emty_cache() both del metax, mask but it doesn't help.

[thsunkid]

Hi @quanvuong,

Had you solved this problem ?
I got one similar when evaluating the baseline model, which caused CUDA error: out of memory due to accumulate the data from each iter.
I used torch 0.4.1 version. Already try to emty_cache() both del metax, mask but it doesn't help.

In my cases, I used torch v0.4.1 instead of v0.3.1 like the author used. I solved my problem by adding with torch.no_grad() during validation because volatile variable in Variable class no longer clear the gradient value, causing accumulated memory in GPU.

[Fangyi-Chen]

Based on my understanding, there are two reasons for the out-of-memory during tuning

1. during the tuning phase, 20 class instead of 15 classes are fed into the re-weighting net. which causes more GPU memory usage.
2. During the tuning phase, for multi-scale training, the input images can be as large as 600+. which leads to dynamic memory usage.

The solution could be 1. decrease the batch size a little bit
2. resize the input image size carefully