Unstable training with keypoint loss about omni3d

[gkioxari]

Hi Johannes.

It's a bit hard for us to advise you when so many things are changing, namely the detector architecture and the dataset.

The fact that 2D detection and 3D detection are unstable on the training set is alarming and I would look closely at your dataset. Is the instability triggered by a learning rate diff (note we use warmup). Do you train with sufficiently large batch sizes, following our recipe? What happens when you turn off all 3D losses and only train with 2D?

Generally speaking it sounds like many things are changing in our model training and also you don't provide how you deviate from our recipe. I would suggest looking closer to your data and ablating various aspects of your model training. Unfortunately, we don't have the capacity to consult you on this one, but I hope my response was helpful.