Stuck in training about diffusiondet

[pierlj]

For me, the reason behind the lock was images with more objects than the model proposal combined with weird pytorch round-off error. Basically, the matcher is not designed to handle more ground truth than queries, in this case, it should always end-up in an infinite loop. However, due to pytorch round-off errors on the cost matrix (matched queries cost is incremented by a 1e6 at each iteration), it finds an exit to the while loop most of the time. In rare cases, it gets stuck in the loop forever.

The solution for me was to prevent having images with too many ground truth, however this is not optimal. You could also increase the number of proposals in the network or rework the matcher.

[pierlj]

Hi, does it crash or freeze completely without any message?

If it freezes, I recently ran into a similar issue. Basically, the training gets stuck in the while loop inside the query-ground truth matching.

[qingqingbaibai]

The program is stuck without error. Maybe it is stuck in the while loop as you said.

[qingqingbaibai]

thanks!