pjrambo / vins-fusion-gpu

This repository is a version of VINS-Fusion with gpu acceleration. It can run on the Nvidia TX2 in realtime

VINS-Fusion-gpu

This repository is a version of VINS-Fusion with GPU acceleration. It can run on Nvidia TX2 in real-time.

1. Prerequisites

The essential software environment is same as VINS-Fusion. Besides, it requires OpenCV cuda version.(Only test it on OpenCV 3.4.1).

2. Usage

2.1 Change the opencv path in the CMakeLists

In /vins_estimator/CMakeLists.txt, change Line 20 to your path.
In /loop_fusion/CmakeLists.txt, change Line 19 to your path.

2.2 Change the acceleration parameters as you need.

In the config file, there are two parameters for gpu acceleration.
use_gpu: 0 for off, 1 for on
use_gpu_acc_flow: 0 for off, 1 for on
If your GPU resources is limitted or you want to use GPU for other computaion. You can set
use_gpu: 1
use_gpu_acc_flow: 0
If your other application do not require much GPU resources, I recommanded you to set
use_gpu: 1
use_gpu_acc_flow: 1
According to my test, on TX2 if you set this two parameters to 1 at the same time, the GPU usage is about 20%.

VINS-Fusion

An optimization-based multi-sensor state estimator

VINS-Fusion is an optimization-based multi-sensor state estimator, which achieves accurate self-localization for autonomous applications (drones, cars, and AR/VR). VINS-Fusion is an extension of VINS-Mono, which supports multiple visual-inertial sensor types (mono camera + IMU, stereo cameras + IMU, even stereo cameras only). We also show a toy example of fusing VINS with GPS. Features:

• multiple sensors support (stereo cameras / mono camera+IMU / stereo cameras+IMU)
• online spatial calibration (transformation between camera and IMU)
• online temporal calibration (time offset between camera and IMU)
• visual loop closure

We are the top open-sourced stereo algorithm on KITTI Odometry Benchmark (12.Jan.2019).

Authors: Tong Qin, Shaozu Cao, Jie Pan, Peiliang Li, and Shaojie Shen from the Aerial Robotics Group, HKUST

Videos:

Related Papers: (papers are not exactly same with code)

• A General Optimization-based Framework for Local Odometry Estimation with Multiple Sensors, Tong Qin, Jie Pan, Shaozu Cao, Shaojie Shen, aiXiv pdf

• A General Optimization-based Framework for Global Pose Estimation with Multiple Sensors, Tong Qin, Shaozu Cao, Jie Pan, Shaojie Shen, aiXiv pdf

• Online Temporal Calibration for Monocular Visual-Inertial Systems, Tong Qin, Shaojie Shen, IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS, 2018), best student paper award pdf

• VINS-Mono: A Robust and Versatile Monocular Visual-Inertial State Estimator, Tong Qin, Peiliang Li, Shaojie Shen, IEEE Transactions on Robotics pdf

If you use VINS-Fusion for your academic research, please cite our related papers. bib

1. Prerequisites

1.1 Ubuntu and ROS

Ubuntu 64-bit 16.04 or 18.04. ROS Kinetic or Melodic. ROS Installation

1.2. Ceres Solver

Follow Ceres Installation.

2. Build VINS-Fusion

Clone the repository and catkin_make:

    cd ~/catkin_ws/src
    git clone https://github.com/HKUST-Aerial-Robotics/VINS-Fusion.git
    cd ../
    catkin_make
    source ~/catkin_ws/devel/setup.bash

(if you fail in this step, try to find another computer with clean system or reinstall Ubuntu and ROS)

3. EuRoC Example

Download EuRoC MAV Dataset to YOUR_DATASET_FOLDER. Take MH_01 for example, you can run VINS-Fusion with three sensor types (monocular camera + IMU, stereo cameras + IMU and stereo cameras). Open four terminals, run vins odometry, visual loop closure(optional), rviz and play the bag file respectively. Green path is VIO odometry; red path is odometry under visual loop closure.

3.1 Monocualr camera + IMU

    roslaunch vins vins_rviz.launch
    rosrun vins vins_node ~/catkin_ws/src/VINS-Fusion/config/euroc/euroc_mono_imu_config.yaml 
    (optional) rosrun loop_fusion loop_fusion_node ~/catkin_ws/src/VINS-Fusion/config/euroc/euroc_mono_imu_config.yaml 
    rosbag play YOUR_DATASET_FOLDER/MH_01_easy.bag

3.2 Stereo cameras + IMU

    roslaunch vins vins_rviz.launch
    rosrun vins vins_node ~/catkin_ws/src/VINS-Fusion/config/euroc/euroc_stereo_imu_config.yaml 
    (optional) rosrun loop_fusion loop_fusion_node ~/catkin_ws/src/VINS-Fusion/config/euroc/euroc_stereo_imu_config.yaml 
    rosbag play YOUR_DATASET_FOLDER/MH_01_easy.bag

3.3 Stereo cameras

    roslaunch vins vins_rviz.launch
    rosrun vins vins_node ~/catkin_ws/src/VINS-Fusion/config/euroc/euroc_stereo_config.yaml 
    (optional) rosrun loop_fusion loop_fusion_node ~/catkin_ws/src/VINS-Fusion/config/euroc/euroc_stereo_config.yaml 
    rosbag play YOUR_DATASET_FOLDER/MH_01_easy.bag

4. KITTI Example

4.1 KITTI Odometry (Stereo)

Download KITTI Odometry dataset to YOUR_DATASET_FOLDER. Take sequences 00 for example, Open two terminals, run vins and rviz respectively. (We evaluated odometry on KITTI benchmark without loop closure funtion)

    roslaunch vins vins_rviz.launch
    (optional) rosrun loop_fusion loop_fusion_node ~/catkin_ws/src/VINS-Fusion/config/kitti_odom/kitti_config00-02.yaml
    rosrun vins kitti_odom_test ~/catkin_ws/src/VINS-Fusion/config/kitti_odom/kitti_config00-02.yaml YOUR_DATASET_FOLDER/sequences/00/ 

4.2 KITTI GPS Fusion (Stereo + GPS)

Download KITTI raw dataset to YOUR_DATASET_FOLDER. Take 2011_10_03_drive_0027_synced for example. Open three terminals, run vins, global fusion and rviz respectively. Green path is VIO odometry; blue path is odometry under GPS global fusion.

    roslaunch vins vins_rviz.launch
    rosrun vins kitti_gps_test ~/catkin_ws/src/VINS-Fusion/config/kitti_raw/kitti_10_03_config.yaml YOUR_DATASET_FOLDER/2011_10_03_drive_0027_sync/ 
    rosrun global_fusion global_fusion_node

5. VINS-Fusion on car demonstration

Download car bag to YOUR_DATASET_FOLDER. Open four terminals, run vins odometry, visual loop closure(optional), rviz and play the bag file respectively. Green path is VIO odometry; red path is odometry under visual loop closure.

    roslaunch vins vins_rviz.launch
    rosrun vins vins_node ~/catkin_ws/src/VINS-Fusion/config/vi_car/vi_car.yaml 
    (optional) rosrun loop_fusion loop_fusion_node ~/catkin_ws/src/VINS-Fusion/config/vi_car/vi_car.yaml 
    rosbag play YOUR_DATASET_FOLDER/car.bag

6. Run with your devices

VIO is not only a software algorithm, it heavily relies on hardware quality. For beginners, we recommend you to run VIO with professional equipment, which contains global shutter cameras and hardware synchronization.

6.1 Configuration file

Write a config file for your device. You can take config files of EuRoC and KITTI as the example.

6.2 Camera calibration

VINS-Fusion support several camera models (pinhole, mei, equidistant). You can use camera model to calibrate your cameras. We put some example data under /camera_models/calibrationdata to tell you how to calibrate.

cd ~/catkin_ws/src/VINS-Fusion/camera_models/camera_calib_example/
rosrun camera_models Calibrations -w 12 -h 8 -s 80 -i calibrationdata --camera-model pinhole

7. Acknowledgements

We use ceres solver for non-linear optimization and DBoW2 for loop detection, a generic camera model and GeographicLib.

8. License

The source code is released under GPLv3 license.

We are still working on improving the code reliability. For any technical issues, please contact Tong Qin <qintonguavATgmail.com>.

For commercial inquiries, please contact Shaojie Shen <eeshaojieATust.hk>.

vins-fusion-gpu's Issues

Using Vins fusion with d435i

Hi @pjrambo @xuhao1 ,
I'm using vins fusion with my realsense d435i camera but I'm getting a problem regarding axis of d435i. In d4345i y is upward and z is forward by default.Now as y is upward and z is forward then the localisation data of vins is also getting wrong because while I'm moving my drone upward the value of -y is increasing and when moving forward the value of z is increasing..So now how i can fix this asix issue??
as there can be two solutions:-
i.)Fiz the by default axis of d435i as per our requirement (y-forward,z-upward)
ii.) Make the vins compatible on the default axis of d435i..
Can you please help how can we do that??

TX2运行该代码上延迟较大的问题

您好，非常感谢您的工作，我将该代码在自己的TX2上使用d455测试后发现，相机640x480分辨率，15fps下能够较好的运行，但是帧率提高到30fps就会有较大的延迟，关于gpu加速的opencv3.4.1等我确认设置正确了，并且监控gpu发现使用gpu和不使用的情况的gpu占有率有20%左右的提升。我尝试降低提取特征点数也不管用，因此想请教下您这个问题

您好，感谢分享，我在TX2上用小觅深度相机有很高的延迟

`%YAML:1.0

#common parameters
#support: 1 imu 1 cam; 1 imu 2 cam: 2 cam;
imu: 1
num_of_cam: 2
imu_topic: "/mynteye/imu/data_raw"
image0_topic: "/mynteye/left/image_raw"
image1_topic: "/mynteye/right/image_raw"
output_path: "/home/tong/output/"

cam0_calib: "left_mei.yaml"
cam1_calib: "right_mei.yaml"
image_width: 752
image_height: 480

Extrinsic parameter between IMU and Camera.

estimate_extrinsic: 0 # 0 Have an accurate extrinsic parameters. We will trust the following imu^R_cam, imu^T_cam, don't change it.
# 1 Have an initial guess about extrinsic parameters. We will optimize around your initial guess.

body_T_cam0: !!opencv-matrix
rows: 4
cols: 4
dt: d
data: [ 4.2812441490024389e-03, -9.9997001507473682e-01,
-6.4528985710044385e-03, 5.2583356071589790e-05,
9.9996900935734523e-01, 4.2384270612576547e-03,
6.6344601088757426e-03, -4.2174706544162562e-02,
-6.6069110351583190e-03, -6.4811023350536514e-03,
9.9995717110239080e-01, 1.9238715201769417e-02, 0., 0., 0., 1. ]

body_T_cam1: !!opencv-matrix
rows: 4
cols: 4
dt: d
data: [ 1.6525177162010074e-03, -9.9998509853672479e-01,
-5.2030654132755290e-03, -2.9251875934051923e-03,
9.9994935393851081e-01, 1.6007594247438028e-03,
9.9361525360214843e-03, 7.4379061731550719e-02,
-9.9276756168115085e-03, -5.2192215666013279e-03,
9.9993709851324453e-01, 1.9302672329335892e-02, 0., 0., 0., 1. ]

#Multiple thread support
multiple_thread: 1
#Gpu accleration support
use_gpu: 1
use_gpu_acc_flow: 1
#feature traker paprameters
max_cnt: 150 # max feature number in feature tracking
min_dist: 30 # min distance between two features
freq: 10 # frequence (Hz) of publish tracking result. At least 10Hz for good estimation. If set 0, the frequence will be same as raw image
F_threshold: 1.0 # ransac threshold (pixel)
show_track: 1 # publish tracking image as topic
flow_back: 1 # perform forward and backward optical flow to improve feature tracking accuracy

#optimization parameters
max_solver_time: 0.04 # max solver itration time (ms), to guarantee real time
max_num_iterations: 8 # max solver itrations, to guarantee real time
keyframe_parallax: 10.0 # keyframe selection threshold (pixel)

#imu parameters The more accurate parameters you provide, the better performance
acc_n: 0.01 # accelerometer measurement noise standard deviation. #0.2 0.04
gyr_n: 0.001 # gyroscope measurement noise standard deviation. #0.05 0.004
acc_w: 0.0001 # accelerometer bias random work noise standard deviation. #0.02
gyr_w: 0.00001 # gyroscope bias random work noise standard deviation. #4.0e-5
g_norm: 9.8 # gravity magnitude

#unsynchronization parameters
estimate_td: 0 # online estimate time offset between camera and imu
td: 0.0 # initial value of time offset. unit: s. readed image clock + td = real image clock (IMU clock)

#loop closure parameters
load_previous_pose_graph: 0 # load and reuse previous pose graph; load from 'pose_graph_save_path'
pose_graph_save_path: "/home/tony-ws1/output/pose_graph/" # save and load path
save_image: 1 # save image in pose graph for visualization prupose; you can close this function by setting 0
`
这是我的配置文件，开启了GPU加速但是似乎没有解决延迟问题，而且随着时间增长内存会爆掉。

Erroneous use of sensor_msgs::ImageConstPtr leads to extravagant memory overflow

In our experiment with Nvidia Jetson TX2, memory utilization of vins-gpu could go up to 85% (as mentioned in #2 ) and would eventually escalate sigkill. After we examined the main entry of ROS node rosNodeTest.cpp, we found you have used queue<sensor_msgs::ImageConstPtr> in line 29,30, which may bring about miserable results such as failure of memory deallocation of sensor_msgs::Image, thus causing memory panic.
After making some minor changes, i.e. using queue<cv::Mat> and convert msgs just in time, this problem can be addressed. Note: extra care should be taken since cv::Mat wouldn't save extra info such as timestamp, which means you have to store it somewhere else manually.
Hope this helps

changing body_T_cam

I have a robot where the camera is mounted on a moving platform and the IMU is attached to a fix part of the rover body. Is there any way to update the relative positions at runtime?

Problems with missing data sets

In the "Problems with missing data sets" section of github, I found that the download link to the Google cloud disk for the dataset you gave is not working, is there another download link I can provide? Thank you very much!

run vins_fusion_gpu with ubuntu20+opencv3.4.13+rosnoetic with problem

could you please tell me the way to sovle it? i met the problem like this:terminate called after throwing an instance of 'cv::Exception'
what(): OpenCV(3.4.13) /home/victor/opencv-3.4.13/modules/core/src/cuda/gpu_mat.cu:116: error: (-217:Gpu API call) unknown error in function 'allocate'.

Update README regarding the installation guide

Issue: Application can't link to the correct OpenCV directory at runtime

Error message after evoking rosrun vins vins_node [config.yaml]:

terminate called after throwing an instance of 'cv::Exception'
  what():  /tmp/binarydeb/ros-kinetic-opencv3-3.3.1/modules/core/include/opencv2/core/private.cuda.hpp:107: error: (-216) The library is compiled without CUDA support in function throw_no_cuda

Aborted (core dumped)

QUESTION ABOUT THE ACCELERATION PARAMETERS

HI @pjrambo ,
could you please tell me in which config file i can change the acceleration parameters ? THANK YOU!

use_gpu and use_gpu_acc_flow parameters are no longer in the config file

I realized that in the past, use_gpu and use_gpu_acc_flow parameters were defined in the config files. But in the latest master, this is no longer the case.

• Where are these two parameters?
• Can you update your README as it is outdated right now?

跑vins的时候出现wait for imu

Aborted (core dumped)

when trying to run it on intelrealsense D435i

enord@enord:~$ rosrun vins vins_node catkin_ws/src/VINS-Fusion-gpu/config/realsense_d435i/realsense_stereo_imu_config.yaml
[ INFO] [1635939173.428655807]: init begins
config_file: catkin_ws/src/VINS-Fusion-gpu/config/realsense_d435i/realsense_stereo_imu_config.yaml
USE_IMU: 1
IMU_TOPIC: /camera/imu
result path /home/enord/output//vio.csv
[ WARN] [1635939173.497895678]: Optimize extrinsic param around initial guess!
camera number 2
[ INFO] [1635939173.523636971]: Unsynchronized sensors, online estimate time offset, initial td: 0
[ INFO] [1635939173.523818733]: ROW: 480 COL: 640
exitrinsic cam 0
-0.00575863 -0.00404633 0.999975
-0.999983 -0.00102246 -0.00576281
0.00104575 -0.999991 -0.00404037
0.0203293 0.00793252 0.00285598
exitrinsic cam 1
-0.00100218 0.000363135 0.999999
-0.999992 -0.00383034 -0.00100078
0.00382998 -0.999993 0.000366971
0.0152858 -0.0524358 0.00869313
set g 0 0 9.805
[ INFO] [1635939173.524544372]: reading paramerter of camera catkin_ws/src/VINS-Fusion-gpu/config/realsense_d435i/left.yaml
OpenCV(3.4.1) Error: Unknown error code -49 (Input file is empty) in cvOpenFileStorage, file /home/enord/opencv-3.4.1/modules/core/src/persistence_c.cpp, line 384
terminate called after throwing an instance of 'cv::Exception'
what(): OpenCV(3.4.1) /home/enord/opencv-3.4.1/modules/core/src/persistence_c.cpp:384: error: (-49) Input file is empty in function cvOpenFileStorage

Aborted (core dumped)

Build failed on TX2

Issue: With TX2, vins_estimator can't build successfully

Error message: C++ internal error

Solution: catkin_make -j1
PLEASE update readme accordingly.

working with compressed image topics

Hello,

How can I make the algorithm subscribe to compressed image topic?
I am using usb_cam node of the ROS.
Also should using compressed images make the process faster or will I lose the same time to decompress the image?

Thank you
Metin

Non real-time performance with Jetson AGX Xavier

I've been working on setting up VINS-Fusion-GPU on Jetson platform and used this docker image for the ease of setup. While testing in real-environments I noticed that it gives non-realtime performance with the following parameters to VINS-fusion-GPU(stereo-inertial):

• max_cnt: 350
• min_dist: 30
• Image rate: 30Hz
• IMU rate: 400Hz

This paper suggests real-time performance with VINS-GPU by using the above mentioned params.
What am I possibly doing wrong? Will running it on docker have an effect on its performance? Also to mention, I've my docker image on SD card since I didn't had enough on-board storage for the same and I've already enabled the power mode by using:

sudo nvpmodel -m0
Enabling jetson clocks

I've tried reducing the input camera data rate to 15Hz, which shows some improvement towards real-time performance but compromises the robustness(more usual failures/drifts).

Where is the config file of GPU settings?

In the config file, there are two parameters for gpu acceleration.
use_gpu: 0 for off, 1 for on
use_gpu_acc_flow: 0 for off, 1 for on

I can't find the config options in *.yaml in config documents.

the master has no realesnse config file

I want to run with D435i like VINS-FUSION, then I found there is no realsense config , but the two branch have, so ,can I use config iin VINS-FUSION ,and then catkin_make to test?

Where is the config file.Where to change the acceleration parameters?

2.2 Change the acceleration parameters as you need。
Where is the config file？
Where to change the acceleration parameters?
Thanks

Running on Jetson Nano? Maxwell 128 cores?

Cool Repo! Do you think VINS-Fusion will run well on a Jetson Nano?

KITTI GPS Fusion (Stereo + GPS) results in skew paths and one is faster than the other

When I try to run the kitti gps fusion test datase t(which I have downloaded from your link) in my jetson tx2 , system results in a funny skew paths - gps based global path and VIO path are in different orientations. Also gps path is updated slower than the other.
What can be the reason?

And one final question, is there a script to calculate rmse between two paths , I also want to try that in euroc results for different configuration parameters.
Thanks for all of this,
Metin

error while catkin build

Errors << vins:make /home/mason/catkin_ws/logs/vins/build.make.001.log
/home/mason/catkin_ws/devel/.private/vins/lib/libvins_lib.so: undefined reference to cv::cuda::SparsePyrLKOpticalFlow::create(cv::Size_<int>, int, int, bool)' /home/mason/catkin_ws/devel/.private/vins/lib/libvins_lib.so: undefined reference to cv:/home/mason/catkin_ws/devel/.private/vins/lib/libvins_lib.so: undefined reference to cv::cuda::/SparsePyrLKOpticalFlow::home/mason/catkin_ws/devel/.private/vins/lib/libvins_lib.so: undefined reference to cv::cuda::SparsePyrLKOpticalFlow::create(cv::Size_, int, int, bool)'
/home/mason/catkin_ws/devel/.private/vins/lib/libvins_lib.so: undefined reference to cv::cuda::createGoodFeaturesToTrackDetector(int, int, double, double, int, bool, double)' :create(cvcuda::createGoodFeaturesToTrackDetector(int, int, double, ::collect2: error: ld returned 1 exit status double, int, bool, double)' Size_<int>, int, int, bool)' /home/mason/catkin_ws/devel/.private/vins/lib/libvins_lib.so: undefined reference to cv::cuda::createGoodFeaturesToTrackDetector(int, int, double, double, int, collect2: error: ld returned 1 exit status
bool, double)'
collect2: error: ld returned 1 exit status
make[2]: *** [/home/mason/catkin_ws/devel/.private/vins/lib/vins/kitti_odom_test] Error 1
make[1]: *** [CMakeFiles/kitti_odom_test.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....
make[2]: *** [/home/mason/catkin_ws/devel/.private/vins/lib/vins/kitti_gps_test] Error 1
make[2]: *** [/home/mason/catkin_ws/devel/.private/vins/lib/vins/vins_node] Error 1
make[1]: *** [CMakeFiles/kitti_gps_test.dir/all] Error 2
make[1]: *** [CMakeFiles/vins_node.dir/all] Error 2
make: *** [all] Error 2
cd /home/mason/catkin_ws/build/vins; catkin build --get-env vins | catkin env -si /usr/bin/make --jobserver-fds=6,7 -j; cd -

I built OpenCV with CUDA using https://github.com/jetsonhacks/buildOpenCVXavier here
and cv2.version returns 3.2.0 for python2 and 3.4.3 for python3.

and I manually built cv_brigde using here https://github.com/ros-perception/vision_opencv
with edited CMakeLists.txt
find_package(OpenCV 3 REQUIRED PATHS /usr/share/OpenCV NO_DEFAULT_PATH
COMPONENTS
opencv_core
opencv_imgproc
opencv_imgcodecs
CONFIG
)
include(/usr/share/OpenCV/OpenCVConfig.cmake)

Is there any mistake of Cmakelist of cv_bridge?
or any other problem of my build process?

Thank you in advance!

The first frame with long processing time

Hi!
When testing the VINS-Fusion-gpu on TX2 nx, I encountered a problem that extracting features on the first frame with the function named goodFeaturesToTrack takes a long time on TX2, which was about 20 to 30 seconds, I wonder how to solve this problem? Thanks.

error while catkin_make

Hello everyone,
So i am trying to test a SLAM algo with OAK camera following the steps in the tuto of: https://www.arducam.com/docs/opencv-ai-kit-oak/performing-location-with-visual-slam/#ftoc-heading-9.
I have a problem in the step 5.5 when building the last 3 targets kitti_gps_test, kitti_odom_test and vins_node. I have as an error :
usr/bin/ld : /home/mehdi/Bureau/stage/depthai/ros_workspace/OAK_Nvidia_Jetson_ROS_SLAM_VINS/VINS_GPU/devel/lib/libvins_lib.so : undefined reference to « cv::cuda::createGoodFeaturesToTrackDetector(int, int, double, double, int, bool, double) »
/usr/bin/ld : /home/mehdi/Bureau/stage/depthai/ros_workspace/OAK_Nvidia_Jetson_ROS_SLAM_VINS/VINS_GPU/devel/lib/libvins_lib.so : undefined reference to « cv::cuda::SparsePyrLKOpticalFlow::create(cv::Size_, int, int, bool) »
collect2: error: ld returned 1 exit status

I tried all solutions on internet but without any result.
I am using OpenCV 3.4.14

Segmentation Fault executing vins_node

Hello!

After compiling VINS-Fusion successfully, on a x86_64 machine, I am trying to execute it using Euroc config file, as so:
rosrun vins vins_node ~/catkin_ws/src/VINS_Fusion/config/euroc/euroc_mono_imu_config.yaml

Execution results in Segmentation fault:
$ rosrun vins vins_node ~/catkin_ws/src/VINS_Fusion/config/euroc/euroc_mono_imu_config.yaml [ INFO] [1600243514.529123941]: init begins config_file: /home/user/catkin_ws/src/VINS_Fusion/config/euroc/euroc_mono_imu_config.yaml Segmentation fault

I have opencv-4.4 built from source.

When trying to debug the executable using gdb, I stumbled upon this error:
Thread 1 "vins_node" received signal SIGSEGV, Segmentation fault. __strlen_avx2 () at ../sysdeps/x86_64/multiarch/strlen-avx2.S:93 93 ../sysdeps/x86_64/multiarch/strlen-avx2.S: No such file or directory.
I wasn't able to identify the root cause of the problem myself.

What could be the problem? Could it be the version of OpenCV?
I have a Jetson Xavier with opencv-4.3 installed on it, and Vins works fine there.

Would be glad for some help,
Thank you for your work!

double free or corruption (out)

When I run vins_node I get the following error.

[ INFO] [1585575375.775787111]: init begins
config_file: ./euroc_mono_imu_config.yaml
USE_IMU: 1
IMU_TOPIC: /imu0
result path /home/tony-ws1/output//vio.csv
[ WARN] [1585575375.785138318]: Optimize extrinsic param around initial guess!
double free or corruption (out)
Aborted (core dumped)

Compute:
TX2
Ubuntu 18.04
Jetpack 4.2
OpenCV 3.4.1

I first installed OpenCV using https://github.com/jetsonhacks/buildOpenCVTX2 and ran into the issue above. I thought the issue might be an incompatibility with the OpenCV build, so I removed OpenCV and installed it with https://blog.csdn.net/L_smartworld/article/details/89843969 but still had the same issue.

I only have this issue with VINS-Fusion-gpu. VINS-Fusion runs.

IMU noise values

Hello,

What are the units of the white noise and random walk noise values that we should enter in the configuration file and how are they implemented correct the IMU measurements in this algorithm?

Thank you,
Metin

Non uniform feature detection on low-contrast frame with local high-contrast areas

I have discovered, that VINS-Fusion-gpu detects less feature points in frames with local high contrast areas. Few feature points are detected in such areas, while the rest of the frame lacks feature points (see the attachment ). After disabling GPU in config file the feature points are detected with a uniform distribution over the whole frame (see the attachment ).
After digging into problem I found out that function cv::goodFeaturesToTrack() considers the input mask region to compute threshold value for the eigenvalues of the corners to be detected. This mask excludes circle regions around the features which are currently being tracked thus allowing threshold to be computed over the rest low-contrast areas. However the GPU-accelerated equivalent function cv::cuda::GoodFeaturesToTrackDetector::detect() does not take the input mask into account when it computes the eigenvalue threshold.
This issue can be easily fixed by adding the fourth parameter mask to the function cuda::minMax() call inside the function cv::cuda::GoodFeaturesToTrackDetector::detect().
In current release of OpenCV 3.4.14 one should modify line 111 of the source file https://github.com/opencv/opencv/blob/3.4.14/modules/cudaimgproc/src/gftt.cpp:
cuda::minMax(eig_, 0, &maxVal);
in the following way:
cuda::minMax(eig_, 0, &maxVal, _mask);
You should initiate a pull request to the OpenCV repository.
Thank you for your work. You have developed an awesome piece of software!

Support for Ceres 2.1.0

The latest version of Ceres comes with CUDA support for DENSE_SCHUR solvers. This might make a significant improvement to the speed of the optimization. Just posting this here in case someone wants to take up the task of getting VINS to work on this version of Ceres. I'd try but my knowledge of Ceres is limited. Any suggestions are welcome, I might give it a shot.

having the error "Not enough features or parallax; Move device around" when running VINS Mono+IMU on tx2

Hello,

I am having the error "Not enough features or parallax; Move device around" when running VINS Mono+IMU on tx2. I have a 1280*720 camera. I have an initial guess of the transformation matrix, even though I try the algorithm with estimate_extrinsic = 0 , 1 or 2 none of them works. I have tried to increase the feature number to 400, it also did not work. Moving the device was also useless for me. I have checked the IMU signal and visualized camera track, camera seems to detect the features well and IMU is working at around 50 hz, increasing it up to 80 is also useless.
This message is sent to ROS_INFO @566th line of the estimator.cpp of the code, I got that the problem is with initialization process but I cannot figure out how to solve it.

Also I have realized that in your rosNodeTest.cpp code, feature_callback function feeds an estimator object with estimator.inputFeature(t,feature) but /feature_tracker/feature topic has no advertiser, I don't think this is the cause of the problem because this topic was also void when I ran euroc examples.

Finally, is just publishin IMU with linear accelerations and angular velocity enough? I could not figure out that should I pre integrate imu and publish it in another topic just for initalization process or not.

Thanks for your help
Metin

error: (-217:Gpu API call) no kernel image is available for execution on the device in function 'caller'

Hi, I installed opencv3.4.1 and cuda10.1 on Jetson NX , but when I compile and configure use_ gpu: 1 when running vins-fusion-gpu, this error will be reported. Please help me solve it. Thank you！！！

terminate called after throwing an instance of 'cv::Exception'
  what():  OpenCV(3.4.10) /home/rb/opencv-3.4.1/modules/cudafilters/src/cuda/row_filter.hpp:169: error: (-217:Gpu API call) no kernel image is available for execution on the device in function 'caller'

d435 shuts down while running on xavier nx with jetpack5.0.1+cuda11.4

My d435 node shut down while running vins as shown below, my platform is xavier nx with jetpack 5.0.1+cuda11.4. I tried to replace mt opencv version, such as 3.4.1, 4.2.0, 4.5.3, but they all didn't work. Have you ran vins on jetpack 5.0.1 and what's you opencv version and cuda version?
error:
malloc(): invalid size (unsorted)
[camera/realsense2 camera manager-2] process has died [pid 407048, exit code -6name:=realsense2 camera manaccmd /opt/ros/noetic/lib/nodelet/nodelet managerlog:=/home/xup250-1/.ros/log/b9dff936-ea84-11ed-95f0-08beac2e2138/camera-ralsenit2-canene-nan2go"i7.ra iog/b9dfr93s-ea84-11ed-9sfo-08beac2e2138/canera-reeralsense2_camera. manager -2* ,log

FeatureTracker::trackImage returns featureFrame with nan values

Dear @pjrambo and @xuhao1 ,

I had the error "Not enough features or parallax; Move device around" when running VINS Mono+IMU on tx2. I have a 1280*720 camera. I have an initial guess of the transformation matrix, even though I try the algorithm with estimate_extrinsic = 0 , 1 or 2 none of them works. I have tried to increase the feature number to 400, it also did not work. Moving the device was also useless for me. I have checked the IMU signal and visualized camera track, camera seems to detect the features well and IMU is working at around 50 hz, increasing it up to 80 is also useless.

Then I have started debugging from estimator::relativePose and I have figured out the NaN values are created when the trackImage() is called. I have realized that FeatureTracker::trackImage returns a featureFrame with NaN values on the X,Y and velocity_x , velocity_y components.
What can be the reason?

thank you for your help,

Sincerely Yours,
Metin

fps setting

Calibration of D435i was carried out using Kalibr.

If I want to proceed vins-fusion to 15fps, would it be better to set D435i to 15fps for calibration?

I would like to know if there is no problem if I proceed with calibration with 30fps and use it with 15fps.

无图像出现

不知道是不是要安装RealSense ROS功能包