Error while running the roslaunch sptam kitti.launch about sptam

[taihup]

Hi @ihtishamaliktk,
Yeah it seems you have a linking issue with g2o. Check: http://gdwarner.blogspot.com.ar/2009/03/c-runtime-symbol-lookup-error.html to understand the error.

Then, if that didn't help try to follow this comment #40 (comment).

What Ubuntu version do you have? remember that S-PTAM was tested in 16.04. It was not tested in other Ubuntu versions.

[ihtishamaliktk]

@taihup Thank you for the prompt response.
Changing eigen 3.3 and compiling g2o with eigen 3.2.10 helped with that issue. Thanks.
Now the process stops with exit code -11 and no LOG file

No handlers could be found for logger "roslaunch"
[start_broadcaster-1] ERROR: unable to configure logging [[Errno 13] Permission denied: u'/home/civit/.ros/log/b1808bc2-741a-11ea-aad1-04ed338e233a/start_broadcaster-1-stdout.log']
process[start_broadcaster-1]: started with pid [23366]
[sptam_path-2] ERROR: unable to configure logging [[Errno 13] Permission denied: u'/home/civit/.ros/log/b1808bc2-741a-11ea-aad1-04ed338e233a/sptam_path-2-stdout.log']
process[sptam_path-2]: started with pid [23372]
process[sptam-3]: started with pid [23384]
[rospack] Unable to create temporary cache file /home/civit/.ros/.rospack_cache.9ayaON: Permission denied
[ INFO] [1585747181.347171063]: Initializing nodelet with 8 worker threads.
  crossCheck: 0
.
.
.
[ INFO] [1585747221.993525320, 1385799339.765692928]: Trying to intialize map...
[ INFO] [1585747222.007269248, 1385799339.765692928]: Map initialized with 130 points.

[sptam-3] process has died [pid 23384, exit code -11, cmd /home/civit/catkin_SPTAM/devel/lib/sptam/sptam_node /stereo/left/image_rect:=/kitti_stereo/left/image_rect /stereo/right/image_rect:=/kitti_stereo/right/image_rect /stereo/left/camera_info:=/kitti_stereo/left/camera_info /stereo/right/camera_info:=/kitti_stereo/right/camera_info __name:=sptam __log:=/home/civit/.ros/log/b1808bc2-741a-11ea-aad1-04ed338e233a/sptam-3.log].
log file: /home/civit/.ros/log/b1808bc2-741a-11ea-aad1-04ed338e233a/sptam-3*.log

I cant locate the log at the mentioned location:
log file: /home/civit/.ros/log/b1808bc2-741a-11ea-aad1-04ed338e233a/sptam-3*.log

Btw , I run the code as :

% To run master
source ~/catkin_SPTAM/devel/setup.bash
sudo -s
roscore

% run on new terminal
source ~/catkin_SPTAM/devel/setup.bash
rosparam set use_sim_time true
roslaunch sptam kitti.launch

% run on new terminal
rosbag play --clock kitti_04.bag

P.S I am using Ubuntu 16.04 with ROS kinetic. The kitti_04.bag is decompressed.

[taihup]

It seems you have a permission issue.
check: https://answers.ros.org/question/206727/permission-denied/
Try do not use sudo -s command.
Also, I recommend to remove build and devel directories from the catkin_ws (in this case catkin_SPTAM) and build it from the scratch.

[ihtishamaliktk]

Thanks for solving the permission issue. It writes the log now.
I previously catkin cleaned and then again built SPTAM after building g2o. On you recommendation, I repeated this.
I figured out that the weird error with exit code of -11 occurred with kitti_04 . When I tried with kitti_00.bag (which i initially avoided downloading due to larger size). It does not give any error.

However, now it does something different.

[ INFO] [1585759819.662155743]: Loop Detector initializing, loading vocabulary
[ INFO] [1585759825.695137228]: S-PTAM stereo node initialized.
[ INFO] [1585759855.110643919, 1385798153.120601342]: init calib
[ INFO] [1585759855.110739438, 1385798153.120601342]: baseline: 0.537166
[ INFO] [1585759855.122437283, 1385798153.130659678]: Trying to intialize map...
[ INFO] [1585759855.132264305, 1385798153.130659678]: Map initialized with 57 points.

**processes fine for a while then the following warning is repeated till the end of sequence playing**

[ WARN] [1585759876.181245747, 1385798174.177015086]: Not enough points for tracking.
[ WARN] [1585759876.286388486, 1385798174.280056494]: Not enough points for tracking.
[ WARN] [1585759876.387078578, 1385798174.381546571]: Not enough points for tracking.
ERROR: There is not any covisibility keyframes observing the tracked points
[ WARN] [1585759876.481096212, 1385798174.491081785]: Not enough points for tracking.
.
.
.
ERROR: There is not any covisibility keyframes observing the tracked points
[ WARN] [1585760489.237793888, 1385798288.821590719]: Not enough points for tracking.
ERROR: There is not any covisibility keyframes observing the tracked points
[ WARN] [1585760489.339935765, 1385798288.923645193]: Not enough points for tracking.
.
.
. User interrupt

One more point, during processing it does not open any window to show tracked points although it was set with -DSHOW_TRACKED_FRAMES=ON

I have attached an interrupted version of Log file here for kitti_00.bag

[taihup]

What computer do you have? CPU and RAM memory?
in any case you can try run the sequence slower, to do this just do:
rosbag play --clock --rate 0.1 <rosbag_file>
Also try to disable the Loop Closure module.

[taihup]

To visualize the tracked frames you have to run in a terminal rqt or rviz and subscribed to the image topic

[ihtishamaliktk]

My laptop is Thinkpad T490 with Intel core i7-8665U processor, 32 GB RAM(16 Onboard+16) and 2GB GPU (Nvidia GeForce MX250 GDDR5)

For Kitt_00:

• With lower rate (.1 ) it processes everything fine (i can view the left frame and path in rviz) till duration 139. Afterwards same previous error (exit code -11).
• With clock rate (.3) it ran till duration 113 (exit code -11).
• With clock rate (.8) it goes till duration 43 (exit code -11).

For Kitt_04:

• With clock rate (.1 ) it ran till 13.2 (exit code -11).

I tried MIT_01_easy but it decided to give something completely different :p

I can turn off the loop closure for now, but for final state I need the loop closure for a fair comparison work.
In the readme, it is written that Profiling can be turned off for improved performance. In such case, is the Log file still generated since that is were the estimated poses are stored, right?

[taihup]

Try to turn off Eigen vectorization, check CIFASIS/dense-sptam#5 (comment).

If you disable profiling, you can still store the publish poses in ROS messages.

[ihtishamaliktk]

Hi again,
I cleaned SPTAM, g2o, ROS and Eigen
I turned off Eigen vectorization and installed eigen 3.2.10
cmake -D CMAKE_CXX_FLAGS="-DEIGEN_DONT_VECTORIZE" ~/catkin_SPTAM/src/eigen-3.2.10/
The built showed that the selected flag was taken into affect. Afterwards I re-installed g2o , ROS and SPTAM.
I tried with turning off the Profiling and Tracking in SPTAM, but that didn't have much effect on it.
Outcome is still (exit code -11) at nearly same duration infact at slightly lower now.

[taihup]

I am sorry but I run out of ideas about what else to check right now, and I don't have the time to debug by myself the code in this moment.

[ihtishamaliktk]

Thank you for your time :)
btw, I tried turning off loop closure. The visual odometry works fine now. Both kitti_04 and kitti_00 ran completely till the end. Seems like the problem is coming from loop closure.

I can't say why the loop closure is causing this, since I

1. Cloned SPTAM repo first
2. then ran git submodule update --init --recursive in SPTAM repo
3. I used git checkout with DBOW2,DLib,OpenGV and DLoop detector
4. Everything compiles fine and SPTAM initializaes the loop detector and loads the vocabulary.
   Any idea why is it affected marginally by clock rate?

[taihup]

you mean the rosbag flag --rate, it slows down the rosbag message rate, given to S-PTAM more time to process the images. It shouldn't be necessary for you computer because it is quite fast. I don't have any idea why LC would be crashing. Maybe, it is compiles with vectorization and it shouldn't. What I mean, maybe you must turn off vectorisation in all the libraries that S-PTAM is used, in a way that everything not using vectorisation.

Moreover, removing Eigen Vectorisation is not desired either because the performance is gonna be dramatically harmed. S-PTAM is not handle well memory alignment and therefore it crashes with other Eigen Libraries or different machine configuration, it is a Bug that we didn't have time to solve.

The right way to solve this is finding the vectorisation bug (memory alignment) and solve it. But as I told you I don't have the time to do this now.

[ihtishamaliktk]

@taihup Hi, here again to bother you a little more :)

I was reading through issue where you suggested to use standalone for more deterministic results. For me the ROS version was also missing poses even with a --rate of 0.05. So I decided to use the standalone as well.

I came accross a few troubles with standalone version, if you can please guide me on those.

With the No Loop Closure, the system runs fine. I see the tracking over images and the program finishes with

.
.
Wait for stop...
-- -- -- WAIT QUEUE - -- --
-- -- -- WAIT JOIN -- -- --
Stop succesfull!

However, I have two problems here:
(1) WARNING tracking is slower than the camera feed by 0.00118795 (s)
(2) The FINAL_FRAME_POSE are not written into the log in standalone directory. The log has ESTIMATED_CAMERA_POSE and REFINED_CAMERA_POSE. I have turned ON profiling and the build verifies that it was ON

The last lines of LOG are related to tracking info.

.
.
1586192558.0590329 xx FindMatchesFrame-match: 0.0005952270000000
1586192558.0590589 XX findMatches-process: 0.0000025510000000
1586192558.0590739 XX findMatches-find: 0.0007955610000000
1586192558.059103 ba refind_newly_made: 0.0186610230000000
1586192558.0877628 ba local_adjust: 0.0286530450000000
1586192558.087929 ba local_save_points: 0.0001567780000000
1586192558.087944 ba local_save_cameras: 0.0000097250000000
1586192558.088006 ba local_handle_bad: 0.0000577920000000
1586192558.0883548 ba remove_bad_points_lock: 0.0000001800000000
1586192558.0883598 ba handleBadPoints: 0.0003499330000000
1586192558.089247 ba totalba: 0.0969066260000000

With Loop closure enabled, it throws up an error, which I presume has the same origin which you explained earlier ?

.
.
Initializing Loop Detector, loading vocabulary
terminate called after throwing an instance of 'std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >'
Aborted (core dumped)

[taihup]

(1) WARNING tracking is slower than the camera feed by 0.00118795 (s)

This means that S-PTAM would lose a frame in the real-time case. In the standalone version S-PTAM is able to process every frame.

2. The FINAL_FRAME_POSE are not written into the log in standalone directory. The log has ESTIMATED_CAMERA_POSE and REFINED_CAMERA_POSE. I have turned ON profiling and the build verifies that it was ON.

In the standalone version, you must take the REFINED_CAMERA_POSE as the final pose. It is not implemented how to add the transformation between baseline and camera system, so it is not possible to compute the FINAL_FRAME_POSE. Remember that standalone was released only for debugging porpoise because it is quite difficult to debug with ROS sometimes.

Yes, the errors seems to be related to how Loop Closure module was compiled and how it interacts with the rest of the S-PTAM modules. It seems LC is using memory alignment (Eigen vectorisation) but the rest of S-PTAM does not. You can try to compile LC module without memory alignment. The same apply to the rest of the S-PTAM dependences (All of them should be compiled with non memory alignment). However, disable memory alignment will harm dramatically the performance of the system.

[ihtishamaliktk]

One more question, the implementation seems to be using too much memory for longer sequences. I have a custom sequence with 20k frames at high resolution (2013 1195). I am using standalone SPTAM without Loop closure in debug (release mode is more problematic). It eats up 32GB memory fairly quickly for some sequences with more discriminate features. I tuned down the number of features to 100 but it buys only a little more time. Any tips? I just need the trajectory from refined poses, not the map (which i assumes is consuming all the memory). Is it possible to turn it off?

[taihup]

mmm Maybe you can try downsampling the images, also depending of the sequence (scene and camera motion), maybe you can change the keyframe selection strategy.

[ihtishamaliktk]

Thank you for the earlier suggestion.
I have faced a few more issues, if you can briefly comments on them it would be help full

(1) Sometimes the process gets killed for no apparent reason. It would give no other error and get immediately killed in the middle of processing.

(2) This one is more concrete. Is the following error related to convergence of solution at the pose estimation? I was able to process that segment of the sequence separately with no error. So, I cant understand if this should have anything to do with bad matches.

int g2o::csparse_extension::cs_cholsolsymb(const cs_di*, double*, const cs_dis*, double*, int*): cholesky failed!
Cholesky failure, writing debug.txt (Hessian loadable by Octave)
[SetJac] infinite jac

[taihup]

I only can give some information about number (2). The error comes from the optimization library g2o. g2o is used in three different places: Tracking (pose estimation, map is not optimizazed only current camera pose), Local Mapping (Local Map and covisibility keyframes are optimized) and Loop Closure (keyframe poses are optimized). So g2o is failing in one of those optimizations procedures.

[ihtishamaliktk]

@taihup Thank you for clarifying all the issues. The info. was really helpful and saved me a lot of time. I have no more questions, so this issue can be closed now. Thanks again.

[taihup]

Your welcome! good to know that it helps :)

[YilmazKadir]

Hello, I am getting the same issue about cholesky failure. Can you please explain how did you solve this problem? I know it has been a long time, but any help would be appreciated.

Cholesky failure, writing debug.txt (Hessian loadable by Octave)
[SetJac] infinite jac

[ihtishamaliktk]

@kdr193 What I noticed was that in some cases the optimizer isn't able to converge (because of the recorded scene). This can happen at either pose estimation, bundle adjustment, or loop closure. In my case, it happened at pose estimation and/or bundle adjustment. Playing with 'nFeatures' and 'minimumTrackedPointsRatio' slightly helps to mitigate it. The idea is to maintain enough valid/good matches between the map and the new scene image so that the optimizer is able to converge to a logical solution.
The response is a bit late but I hope it helps.

[YilmazKadir]

Thanks a lot for the explanation. I will try it out.

@kdr193 What I noticed was that in some cases the optimizer isn't able to converge (because of the recorded scene). This can happen at either pose estimation, bundle adjustment, or loop closure. In my case, it happened at pose estimation and/or bundle adjustment. Playing with 'nFeatures' and 'minimumTrackedPointsRatio' slightly helps to mitigate it. The idea is to maintain enough valid/good matches between the map and the new scene image so that the optimizer is able to converge to a logical solution.
The response is a bit late but I hope it helps.