the accuracy of experimental results about ccm_slam

[patriksc]

Hello, thanks a lot for your kind feedback on CCM-SLAM.

• Regarding the save-load functionality: Yes, you can directly load the saved map again into CCM-SLAM, as described here: https://github.com/VIS4ROB-lab/ccm_slam/tree/devel#44-saving-and-loading-maps

• Regarding the accuracy of the estimate: The accuracy might vary a bit on different machines, or there might be also be outliers runs, but you should not consistently get a trajectory error in this range. However, I did never use EVO myself, so I cannot say whether it might be something related to the evaluation tool itself (but it shouldn't). Are you running all 4 agents in parallel on a single machine? This might overload a "normal" consumer machine, and result in a high error in the end. It would be great if you could provide a some more information on your experimental setup / process, maybe this helps to find an explanation for the high error.

[RuiYangQuan]

I have similar doubts about the global trajectory posture，are the KF_GBA_0.txt, KF_GBA_1.txt, KF_GBA_2.txt, and KF_GBA_3.txt output by the system the result of the server's global optimization of the pose?Or is there another way to get global trajectories？

[Yuan-TS]

Hello, thanks a lot for your kind feedback on CCM-SLAM.

• Regarding the save-load functionality: Yes, you can directly load the saved map again into CCM-SLAM, as described here: https://github.com/VIS4ROB-lab/ccm_slam/tree/devel#44-saving-and-loading-maps
• Regarding the accuracy of the estimate: The accuracy might vary a bit on different machines, or there might be also be outliers runs, but you should not consistently get a trajectory error in this range. However, I did never use EVO myself, so I cannot say whether it might be something related to the evaluation tool itself (but it shouldn't). Are you running all 4 agents in parallel on a single machine? This might overload a "normal" consumer machine, and result in a high error in the end. It would be great if you could provide a some more information on your experimental setup / process, maybe this helps to find an explanation for the high error.

Thank you very much for your patient recovery, which helped me a lot. Regarding the experimental configuration, I did run 4 agents and 1 server on one machine, and used rviz to display the results in real time. Later, I tried to turn off the display function of rviz, and the magnitude of the experimental results was consistent with the results in the paper, so I guess it might still be a computer performance problem. Because of your help, this problem has been solved smoothly. Thanks again to you and your team!

[Yuan-TS]

I have similar doubts about the global trajectory posture，are the KF_GBA_0.txt, KF_GBA_1.txt, KF_GBA_2.txt, and KF_GBA_3.txt output by the system the result of the server's global optimization of the pose?Or is there another way to get global trajectories？

After testing and comparison, I think these files are the result of global optimization output by the system.

[RuiYangQuan]

I have similar doubts about the global trajectory posture，are the KF_GBA_0.txt, KF_GBA_1.txt, KF_GBA_2.txt, and KF_GBA_3.txt output by the system the result of the server's global optimization of the pose?Or is there another way to get global trajectories？

After testing and comparison, I think these files are the result of global optimization output by the system.

Thank you for your reply, but I may be slow, these files should be the optimization results of each CIent, so how to use tools like EVO to measure the accuracy of the whole system, please help me, thank you very much ！！

[patriksc]

@Yuan-TS Thanks for your feedback. Regarding your experimental setup: when you run simultaneously 4 agents, plus the server, on a single machine, your machine basically has to support 4 individual VIO algorithms and a global SLAM back-end, which is a usually too much load for a single consumer PC (like my Thinkpad), which then results in a degradation of the estimation performance. However, it's also more convenient for functional tests than using multiple PCs simultaneously. A good direction to mitigate the effect, if you want to keep running in parallel on one machine, is to reduce the playback speed of the rosbag file (like rosbag play mybag.bag -r 0.25).

@RuiYangQuan You need to merge the individual trajectory data (KF_GBA_0.txt, KF_GBA_1.txt, ...) into a single file, and do the same with the relevant groundtruth data that comes with the EuRoC dataset. Then you can evaluate using evo, e.g. with evo_ape euroc my_estimate.csv my_gt_data.csv -vas. However, note that evo cannot read the files from CCM-SLAM directly, you have to convert them to TUM format, unless you pull our recent update.

Since we think it would be beneficial for more users to be able to use evo out-of-the-box, we have changed the format of the trajectory files to fit the TUM format, so that evo can be directly process them.

[patriksc]

Problem seems resolved - closing the issue.