kitti_map.py axis order issue about cmrnet

[LeonardPatrick]

I suppose a classical pipeline should be: you know your ego pose in lidar frame, crop the local_map in lidar frame, do offset in lidar frame and then use velo2cam2 to transform the points in lidar frame to camera model, then project, x/z->u, y/z->v, done. However, this setting is not work. I am really curious about the difference.

[cattaneod]

Hi @LeonardPatrick ,

the axis order is just a convention, and should not impact the performance. If you prefer to use a different axis order, you need to modify the code that projects lidar points in the camera: https://github.com/cattaneod/CMRNet/blob/master/camera_model.py#L18

To test that the projection is correct, you can set max_r and max_t to zero (just for debugging purpose), and visualize the RGB image and the lidar projection (rgb_img and refl_img in https://github.com/cattaneod/CMRNet/blob/master/main_visibility_CALIB.py#L90), and check that they visually align. If they do not align, there is something wrong in your projection.

Let me know if you are able to solve the issue.

[LeonardPatrick]

Thank you so much for your nice explaination. I confirm my projection is correct and aligned visually in the following description: 'I suppose a classical pipeline should be: you know your ego pose in lidar frame, crop the local_map in lidar frame, do offset in lidar frame and then use velo2cam2 to transform the points in lidar frame to camera model, then project, x/z->u, y/z->v, done. However, this setting is not work. I am really curious about the difference.' However, the training is not convergent in the new setting. I guess the difference is that the overlap is smaller in the new setting: do pose offset firstly then velo2cam2. I think your pose offset RT is not R_10·T_2, but R_order * R_10·T_2* R_order_inverse. My key claim is: When we use CMRNet in real-world scenario, we have to recover the pose in lidar frame or world frame , not in a camera frame(plus 2 times order change).

[cattaneod]

Thank you so much for your nice explaination. I confirm my projection is correct and aligned visually

Can you share the lidar overlayed with the camera in the ground truth?

the training is not convergent in the new setting.

If the only difference is the axis alignment, then I'm pretty confident that the performance should not change. I suppose that either the projection or the ground truth are not correct.

To compare, I try to add pose offset RT in lidar frame

What do you mean exactly? which pose RT are you adding? and to what?

I think your pose offset RT is not R_10·T_2, but R_order * R_10·T_2* R_order_inverse.

As the random augmentation is applied on each axis independently, the result of the two should be similar.

When we use CMRNet in real-world scenario, we have to recover the pose in lidar frame or world frame , not in a camera frame(plus 2 times order change).

This is not exactly true, the goal of the method is to recover the pose of the camera, as the use case is to use CMRNet on robots/car without a LiDAR sensor. If you need to recover the LiDAR pose, you can simply apply cam2velo to the camera pose.

Unfortunately, I cannot test it myself at the moment, as I'm busy with other papers.

[LeonardPatrick]

Thank you so much for your confidence, so I try to focus on the dimension. I just think your order change make the z direction down, which will be confusing. Anyway, I find the reason of performance decay: the offset pose in z direction has a upper limit of 1m, but in my case, the z direction is up. So it should be lower limit of -1. It works. I am a little corious about the limit, why is the overlapping so small when z is outside [-2, 1]. Thank you so much, look forward to your new CMRNet++ Paper.

[cattaneod]

I'm glad that you solved the issue.

The problem when the Z is too low (or too high, depending on the axis representation), is that the "virtual" camera will end below the road surface, and therefore the lidar projection will be only a single plane in the sky, making the matching very hard, if not impossible.

[LeonardPatrick]

Thank you very much, it makes sense to me for the sky part.