training error about GC？ about fcos

[ZhengyuZhang96]

by the way ,The first time i encountered this error when training to step=1400.
how can i solve it ? thanks !

[tianzhi0549]

@studyharderer Please try to set DATALOADER.NUM_WORKERS as 0.

[ZhengyuZhang96]

@tianzhi0549 Thank you very much for your reply ! After setting DATALOADER.NUM_WORKERS=0 , it runs to iters=3340 and reports the following error:

Fatal Python error: GC object already tracked

Thread 0x00007f52917fe700 (most recent call first):

Thread 0x00007f52a6cbe700 (most recent call first):

Thread 0x00007f52a813e700 (most recent call first):

Thread 0x00007f5291fff700 (most recent call first):

Current thread 0x00007f5388734740 (most recent call first):
File "/home/zzy/anaconda3/envs/FCOS/lib/python3.7/site-packages/torchvision/datasets/coco.py", line 115 in getitem
File "/home/zzy/fcos/FCOS/maskrcnn_benchmark/data/datasets/coco.py", line 67 in getitem
File "/home/zzy/anaconda3/envs/FCOS/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 529 in
File "/home/zzy/anaconda3/envs/FCOS/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 529 in next
File "/home/zzy/fcos/FCOS/maskrcnn_benchmark/engine/trainer.py", line 56 in do_train
File "/home/zzy/fcos/FCOS/tools/train_net.py", line 73 in train
File "/home/zzy/fcos/FCOS/tools/train_net.py", line 167 in main
File "/home/zzy/fcos/FCOS/tools/train_net.py", line 174 in

What does this error mean ? how can i solve it ?

[tianzhi0549]

@studyharderer I am not sure why this error happens on your machine. Did you change any of the code? Can you try to use a larger batch size such as 4?

[ZhengyuZhang96]

@tianzhi0549 I think I might have found the problem. When I started training, the CPU usage of the server reached more than 400% ...... And I used one 1080ti to take up 60%. How do I reduce the usage of cpu and convert into gpu?
I found that the model in your code uses model.to(device) to use gpu. Is the training data in gpu or cpu? I didn't understand your code about dataloader.

[tianzhi0549]

@studyharderer In my opinion, I don't think the error has to do with the CPU or GPU usages. Do your server has limited host memory?