Query Regarding Translation Loss Convergence in Real Robot Training about rvt

[imankgoyal]

Hi,

Thanks for your kind words and interest in our work.

Q: When training on real robot data, did you start the training from scratch or was there any preliminary training involved using simulations?
A: We train on real robot data from scratch.

Q: During your experiments with real robot data, did you observe a similar convergence pattern between translation and rotation?
A: This is a great question! Yes, we observed a similar convergence pattern, so what you observe is typical. I would strongly recommend visualizing the predicted translation heatmap on top of virtual images of some train as well as unseen samples. My guess would be that you would observe sharp heatmap predictions. If it is so, it would suggest that the network has converged well and the seemingly (explained below) higher value of the translation loss is not an issue.

Unlike typical losses, which go to 0, the translation loss in RVT does not go to zero but hovers around 3.8 ( probably less for better-trained networks) for an image resolution ~220ish and some other parameters like gt_hm_sigmae. This is because, in RVT, translation loss is the sum of heatmap prediction loss for each virtual image. Predicting the ground-truth heatmap exactly requires predicting the keypoint location on the virtual image with sub-pixel accuracy. As the 3D ground-truth keypoint location can be mapped as a continuous 2D pixel location. This is just a design choice, we could have chosen to map to a discrete pixel location. Since, a network incurs a large loss value even for predicting the keypoint apart by <1 pixel distance, the loss values are large.

[MrTooOldDriver]

Many thanks for your reply! Indeed visualisation does show very reasonable results. Thank you so much!