Comments (7)
I suppose a classical pipeline should be: you know your ego pose in lidar frame, crop the local_map in lidar frame, do offset in lidar frame and then use velo2cam2 to transform the points in lidar frame to camera model, then project, x/z->u, y/z->v, done. However, this setting is not work. I am really curious about the difference.
from cmrnet.
Hi @LeonardPatrick ,
the axis order is just a convention, and should not impact the performance. If you prefer to use a different axis order, you need to modify the code that projects lidar points in the camera: https://github.com/cattaneod/CMRNet/blob/master/camera_model.py#L18
To test that the projection is correct, you can set max_r
and max_t
to zero (just for debugging purpose), and visualize the RGB image and the lidar projection (rgb_img
and refl_img
in https://github.com/cattaneod/CMRNet/blob/master/main_visibility_CALIB.py#L90), and check that they visually align. If they do not align, there is something wrong in your projection.
Let me know if you are able to solve the issue.
from cmrnet.
Thank you so much for your nice explaination. I confirm my projection is correct and aligned visually in the following description: 'I suppose a classical pipeline should be: you know your ego pose in lidar frame, crop the local_map in lidar frame, do offset in lidar frame and then use velo2cam2 to transform the points in lidar frame to camera model, then project, x/z->u, y/z->v, done. However, this setting is not work. I am really curious about the difference.' However, the training is not convergent in the new setting. I guess the difference is that the overlap is smaller in the new setting: do pose offset firstly then velo2cam2. I think your pose offset RT is not R_10·T_2, but R_order * R_10·T_2* R_order_inverse. My key claim is: When we use CMRNet in real-world scenario, we have to recover the pose in lidar frame or world frame , not in a camera frame(plus 2 times order change).
from cmrnet.
Thank you so much for your nice explaination. I confirm my projection is correct and aligned visually
Can you share the lidar overlayed with the camera in the ground truth?
the training is not convergent in the new setting.
If the only difference is the axis alignment, then I'm pretty confident that the performance should not change. I suppose that either the projection or the ground truth are not correct.
To compare, I try to add pose offset RT in lidar frame
What do you mean exactly? which pose RT are you adding? and to what?
I think your pose offset RT is not R_10·T_2, but R_order * R_10·T_2* R_order_inverse.
As the random augmentation is applied on each axis independently, the result of the two should be similar.
When we use CMRNet in real-world scenario, we have to recover the pose in lidar frame or world frame , not in a camera frame(plus 2 times order change).
This is not exactly true, the goal of the method is to recover the pose of the camera, as the use case is to use CMRNet on robots/car without a LiDAR sensor. If you need to recover the LiDAR pose, you can simply apply cam2velo to the camera pose.
Unfortunately, I cannot test it myself at the moment, as I'm busy with other papers.
from cmrnet.
Thank you so much for your confidence, so I try to focus on the dimension. I just think your order change make the z direction down, which will be confusing. Anyway, I find the reason of performance decay: the offset pose in z direction has a upper limit of 1m, but in my case, the z direction is up. So it should be lower limit of -1. It works. I am a little corious about the limit, why is the overlapping so small when z is outside [-2, 1]. Thank you so much, look forward to your new CMRNet++ Paper.
from cmrnet.
I'm glad that you solved the issue.
The problem when the Z is too low (or too high, depending on the axis representation), is that the "virtual" camera will end below the road surface, and therefore the lidar projection will be only a single plane in the sky, making the matching very hard, if not impossible.
from cmrnet.
Thank you very much, it makes sense to me for the sky part.
from cmrnet.
Related Issues (20)
- How can i run this project on RTX3090? HOT 13
- Problem with LocalMaps Generation HOT 20
- Training fails after Epoch 0 HOT 6
- Training fails after epoch 0 HOT 2
- Problem with evaluation script HOT 10
- About model performance HOT 3
- Using a different data set for training and evaluation HOT 12
- RuntimeError: CUDA error: no kernel image is available for execution HOT 1
- Some hints needed on converting to online use HOT 10
- Line of code seems incorrect HOT 1
- Some operations seem difficult to follow HOT 3
- How can I apply the error_tr and error_rot for pose ? HOT 2
- Is there any specifc format for the KITTI extrinsics ?
- Ground truth test pose is required HOT 1
- Occlusion filtering (train vs eval show) HOT 2
- ValueError: mathutils.Euler(): invalid euler order 'XYZ' HOT 2
- When i run kitti_maps.py, it displays "killed". HOT 3
- Does CMRNet++ plan to open source? HOT 1
- Evaluated Error Result issue
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from cmrnet.