specarmi / thermal_superpoint_slam Goto Github PK

I am testing the code with different rosbag I have from thermal cameras. In all of them, it seems to detect the features, but when using the SuperPoint SLAM there is a moment where it gets stuck in "intializing..." after catching some features.

Could you help me with this? Thanks

Hi author!

I was trying to use SuperPoint SLAM runs offline on precomputed keypoints and descriptors. But after I tried to run it on KITTI datasets grayscale 09, some problem happened.

I followed your tutorial and use :
python utils/generate_keypts_and_desc.py ~/catkin_ws/src/Thermal_SuperPoint_SLAM/trained_networks/superpoint_thermal/superPointNet_170000_checkpoint.pth.tar ~/datasets/SLAM/KITTI/grayscale/09/image_0/ RGB_Feat_and_Descriptors
the model superPointNet_170000_checkpoint.pth.tar from here.
then it got some .yaml files in RGB_Feat_and_Descriptors from index 1 to 1951.

I also tried use model superPointNet_50000_checkpoint.pth.tar (from here(trained by KITTI?)) and model superPointNet_100000_checkpoint.pth.tar (which I trained on FLIR ADAS dataset by myself followed your tutorial) got feature_50000_0 and feature_0

But after I use :
./thirdparty/SuperPoint_SLAM/Examples/Monocular/mono_kitti vocabularies/superpoint_voc.yml.gz thirdparty/SuperPoint_SLAM/Examples/Monocular/KITTI04-12.yaml ~/datasets/SLAM/KITTI/grayscale/09/ ~/datasets/SLAM/KITTI/grayscale/09/RGB_Feat_and_Descriptors/
(the vocabulary superpoint_voc.yml.gz come from here)
the program crashed:

Estimated frame rate 10
Creating orb-slam system ...

ORB-SLAM2 Copyright (C) 2014-2016 Raul Mur-Artal, University of Zaragoza.
This program comes with ABSOLUTELY NO WARRANTY;
This is free software, and you are welcome to redistribute it
under certain conditions. See LICENSE.txt.

Input sensor was set to: Monocular

Loading ORB Vocabulary. This could take a while...

Camera Parameters: 
- fx: 707.091
- fy: 707.091
- cx: 601.887
- cy: 183.11
- k1: 0
- k2: 0
- p1: 0
- p2: 0
- fps: 10
- color order: RGB (ignored if grayscale)

ORB Extractor Parameters: 
- Number of Features: 2000
- Scale Levels: 4
- Scale Factor: 1.2
- Initial Fast Threshold: 0.015
- Minimum Fast Threshold: 0.007

-------
Start processing sequence ...
Images in the sequence: 1591

desc size: 2176 256
mvKeys.size(): 2176
desc size: 2157 256
mvKeys.size(): 2157
start to intialize...
desc size: 2175 256
mvKeys.size(): 2175
start to intialize...
intializing...
New Map created with 722 points
buildSystem(): NaN within Jacobian for edge 0x555baed22340 for vertex 0
buildSystem(): NaN within Jacobian for edge 0x555baed24980 for vertex 0
buildSystem(): NaN within Jacobian for edge 0x555baed25520 for vertex 0
......
buildSystem(): NaN within Jacobian for edge 0x555baed2fdc0 for vertex 0
buildSystem(): NaN within Jacobian for edge 0x555baed305e0 for vertex 0
buildSystem(): NaN within Jacobian for edge 0x555baed30960 for vertex 0
Segmentation fault (core dumped)

So I tried to use do vocabulary by myself, I used
./thirdparty/DBoW2/build/build_superpt_vocab ~/datasets/SLAM/KITTI/grayscale/09/feature_50000_0/ got superpt_KITTI_voc.yml.gz

./thirdparty/DBoW2/build/build_superpt_vocab ~/datasets/SLAM/KITTI/grayscale/09/feature_0/ got superpt_voc.yml.gz

But whatever I tried
./thirdparty/SuperPoint_SLAM/Examples/Monocular/mono_kitti vocabularies/superpt_KITTI_voc.yml.gz thirdparty/SuperPoint_SLAM/Examples/Monocular/KITTI04-12.yaml ~/datasets/SLAM/KITTI/grayscale/09/ ~/datasets/SLAM/KITTI/grayscale/09/feature_50000_0/
or
./thirdparty/SuperPoint_SLAM/Examples/Monocular/mono_kitti vocabularies/superpt_voc.yml.gz thirdparty/SuperPoint_SLAM/Examples/Monocular/KITTI04-12.yaml ~/datasets/SLAM/KITTI/grayscale/09/ ~/datasets/SLAM/KITTI/grayscale/09/feature_0/
(also tried vocabularysuperpt_thermal.yml.gz), the program all ran 1 frame and then keeping stuck forever:

Estimated frame rate 10
Creating orb-slam system ...

ORB-SLAM2 Copyright (C) 2014-2016 Raul Mur-Artal, University of Zaragoza.
This program comes with ABSOLUTELY NO WARRANTY;
This is free software, and you are welcome to redistribute it
under certain conditions. See LICENSE.txt.

Input sensor was set to: Monocular

Loading ORB Vocabulary. This could take a while...

Camera Parameters: 
- fx: 707.091
- fy: 707.091
- cx: 601.887
- cy: 183.11
- k1: 0
- k2: 0
- p1: 0
- p2: 0
- fps: 10
- color order: RGB (ignored if grayscale)

ORB Extractor Parameters: 
- Number of Features: 2000
- Scale Levels: 4
- Scale Factor: 1.2
- Initial Fast Threshold: 0.015
- Minimum Fast Threshold: 0.007

-------
Start processing sequence ...
Images in the sequence: 1591

desc size: 1100 256
mvKeys.size(): 1100
desc size: 1037 256
mvKeys.size(): 1037
start to intialize...
desc size: 1043 256
mvKeys.size(): 1043
start to intialize...
intializing...

Have you ever encountered the same problem? Could you please give me some suggestions to solve these problem?
Thank you anyway:pray:

Hello, author:

I am trying to test your algorithm, but I have encountered some problems temporarily.

I noticed in your video that the sequence you used in the test came from UM Ford Center for Autonomous Vehicles.

But I did not find the relevant sequence in UM Ford Center for Autonomous Vehicles.

If possible, could you please provide the download link of the relevant image sequence or other relevant links?

Best wishes.

Hello author!

i was trying to test your algorithm.

but i have a problem to run kitti example.

./thirdparty/SuperPoint_SLAM/Examples/Monocular/mono_kitti ./superpt_voc.yml.gz thirdparty/SuperPoint_SLAM/Examples/Monocular/KITTI03.yaml ../../dataset/data_odometry_gray/dataset/sequences/03/ ../../dataset/data_odometry_gray/dataset/sequences/03/RGB_Feat_and_Descriptors/

above command is that I followed your suggestion.

I follow your tutorial to test ORB-SLAM2 and then ORB SLAM result is fine

but Super Point rgb SLAM is not working.

did i miss something?

The camera trajectory is not working like ORB-SLAM2.

I made a 'superpt_voc.yml.gz' file using only equences/03/image0

best regard!