fixstars / libsgm Goto Github PK

Hi,
First of all thank you for this code. This is the best gpu implementation of SGM I found so far. When I want to execute Reprojection project, but an error occurred,"camera.xml read faild",The cameras000000009.xml I use is downloaded from http://www.6d-vision.com/scene-labeling.However,this mistake still occurs? My computer installed the VS2015+CUDA8.0+Opencv3.1.0, So How should I do? Could you give me some advice or explain how to read camera.xml files?
Look forward to your reply.

I am trying to launch SGM-algorithm with default NUM_PATH and it works fine. But I have noticed that buffer_size in path_aggregation.cu (53 string) depends on NUM_PATH, MAX_DISPARITY. So I want to change NUM_PATH value from 8 to 4 for reduction occupied device memory. After changing this variable program is crashing: segmentation fault. There are ways for changing NUM_PATHS and after SGM will works fine? Thank you very much!

Best Regards,

Hello!
I'm trying to develop cuda with CMakeLists.txt! Which Ide you recommend? I have try Clion and eclipse, but they are difficult to use.

I have found it that the recent version add the min_disp param,but I don't understand what it is for.Could you please explain it?

Hello! I am interested in obtain a smoother disparity map for point cloud generation. How can I apply a filtering / smoothing operation in libSGM (for example, similar to OpenCV's DisparityWLSFilter)? Which function / parameter should I call / tweak to obtain the smoother disparity?

Thanks!

Hi, I have some knowledge on how sub-pixel accuracy works and its implementation, but is it worth the effort to do so. Cause what are the actual use cases it might be needed for.

Thanks and Regards,
Mohanenv B

Hi ,Can anybody please specify the reference datasets,the link posted in the repository doesn't contain the datasets.

When i make the project, there's some error, as follow:

Linking CXX executable stereo_test
../../src/libsgm.so: undefined reference to nppiFilterMedianGetBufferSize_16u_C1R' ../../src/libsgm.so: undefined reference to nppiFilterMedian_16u_C1R'
collect2: error: ld returned 1 exit status
sample/image/CMakeFiles/stereo_test.dir/build.make:107: recipe for target 'sample/image/stereo_test' failed
make[2]: *** [sample/image/stereo_test] Error 1
CMakeFiles/Makefile2:140: recipe for target 'sample/image/CMakeFiles/stereo_test.dir/all' failed
make[1]: *** [sample/image/CMakeFiles/stereo_test.dir/all] Error 2
Makefile:127: recipe for target 'all' failed
make: *** [all] Error 2

My environment is ubuntu 16.04, cuda 9.1, thank you!

When I want to make files in /sample/image/ or /sample/movie/, I get the following errors in order. I use CUDA 10 and CMake 3.14.

CMakeFiles/stereo_test.dir/build.make:62: recipe for target 'CMakeFiles/stereo_test.dir/stereosgm_image.cpp.o' failed
make[2]: *** [CMakeFiles/stereo_test.dir/stereosgm_image.cpp.o] Error 1
CMakeFiles/Makefile2:72: recipe for target 'CMakeFiles/stereo_test.dir/all' failed
make[1]: *** [CMakeFiles/stereo_test.dir/all] Error 2
Makefile:83: recipe for target 'all' failed
make: *** [all] Error 2

CMakeFiles/stereo_movie.dir/build.make:62: recipe for target 'CMakeFiles/stereo_movie.dir/stereosgm_movie.cpp.o' failed
make[2]: *** [CMakeFiles/stereo_movie.dir/stereosgm_movie.cpp.o] Error 1
CMakeFiles/Makefile2:72: recipe for target 'CMakeFiles/stereo_movie.dir/all' failed
make[1]: *** [CMakeFiles/stereo_movie.dir/all] Error 2
Makefile:83: recipe for target 'all' failed
make: *** [all] Error 2

Are the errors related to the version of cuda or CMake?

Hi,
First of all thank you for this code. This is the best gpu implementation of SGM I found so far and I successfully implemented it in my (ROS) code.
However, the result is a bit noisy, when I put my hand too close to the stereo camera, a lot of arbitrary disparities appear. In the Opencv CPU implementation I solved this by increasing the uniqueness ratio.
I know that I can change the uniqueness parameter but that does not seem to have any effect. I tried a lot of (extreme) values but the result is always the same. Am I doing something wrong or is this a bug?

How I initialize the SGM parameters:
sgm::StereoSGM::Parameters m_SGBMParam;
m_SGBMParam.P1 = 10;
m_SGBMParam.P2 = 120;
m_SGBMParam.uniqueness = 0.95f;
m_SGBMParam.subpixel = true;

How I create the SGM object:
sgm::StereoSGM* m_LeftMatcher_SGBM;
m_LeftMatcher_SGBM = new sgm::StereoSGM(m_ImageSizeNew.width, m_ImageSizeNew.height, 64, 8, 16, sgm::EXECUTE_INOUT_CUDA2CUDA, m_SGBMParam);

Kind regards

The Disparity from libSGM, when projected in a point cloud the output contains noise like a pyramid in front of the camera covering the entire scene or the edges of the of the object seem to replicate multiple times in various disparity levels.

I have some knowledge on how sgm works and for post-processing I have been looking into Nerian's post-processing Steps and Hirshmullers Disparity Refinement Steps to improve the disparity map.

I was able to remove the noise (mentioned above) using a speckle filter which retained the correct data but at the cost of making it sparse almost losing more than 50% of the data in some scenes.

And looking at the other post-processing steps like Uniqueness check, Consistency check, Filtering of untextured image areas, Noise reduction seems to make the disparity even sparser and applying an optimal hole/gap filling technique wouldn't be enough I think.

I know SGM can do it better it even won 11th place in the ROB 2018.

So do you guys have any idea on

1. How can I improve the disparity accuracy without making it sparse, what are the Pre/Post processing steps can be implemented to further improve it?
2. How to include sub-pixel accuracy, any plans on including it?

Thanks and Regards,
Mohanen B

libSGM/sample/movie/cuda_gl.cu(26): error: no instance of overloaded function "surf2Dwrite" matches the argument list argument types are: (uint16_t, cudaSurfaceObject_t, unsigned long, const int)

I am wondering if i don't rectify the stereo images, will it influence the cost calculation? because the orientation of the real matched points is very likely not the same, there is always a rotation or other transformation in the surounding content of the real matched points. Additionally, in this configuration, the MAX_DISPARITY might be the width of the right image, which could be over 1000 pixels, because the right image might actually be in the left of the refference image, the global memory could not be enough, but we could use tiling of the refference image to deal with it. However, the shared memory usage is very diffcult to eliminate, can anyone give me some advice to handle it?

Hi, it is normal when I execute this program ./stereo_test on PC. But there is an error on Nvidia TX2.
CUDA ERROR #4 (using FilterMedian kernel #2): unspecified launch failure stereo_test: /home/nvidia/newlibSGM/libSGM/src/median_filter.cu:31: void sgm::details::median_filter(const uint16_t*, uint16_t*, void*, int, int): Assertion status == 0' failed. Aborted (core dumped)`
This mistake confused me for a long time. Looking forward to your reply.

Hi, thank you for your perfect code for stereo matching. In your code, the max disparity is set to 128(default), but I can not find where is the min disparity. Could you explain it? Thank you.

Regards

Hi, @8BitCatJQW
Thank you for using libSGM.
Where was this issue occurred in? Our sample or your repository?
We would apprecite if you tell us more details.

Originally posted by @8BitCatJQW in #46 (comment)

Hi,

I was trying to understand the implementation of SGM in the code. Although the cuda stuffs are doing their job perfectly, I have the following question :
Why the indexing in some kernels pretty complex? For eg, the check_consistency_kernel has neat and understandable indexing where the kernel configuration is fully dependent on image size. Whereas winner_takes_all_kernel does not have indexing with image rows, columns but they use warp-id and lane-ids. Is this done for some kind of optimizations or is it over-engineered?

I can map the theoretical concept of SGM with the implementation but the indexing part is a bit hard to understand especially wherever warp-level primitives are used.

Cheers.

Any examples or reference codes?

Hi,
OpenCV uses 4 times square of SAD window size as p1, and 8*p1 as p2. I think this is easy to understand: p1 threshold is within each SAD window (so total square of SAD window size of pixels) on average each pixel differs by 4.
For census, each pixel generates 31 bits so the cost is mostly 31. This is a much smaller range than opencv's p1 threshold to adjust.
In your code, p1 is 10 and p2 is 120. I wonder what these values come from, especially p2, seems to be too big? In detail, initially, L(p-r,d) is mostly 31, L(p-r, d-1) is mostly 31+p1). I understand that L would might become bigger as the DP goes, but how 120 is get, just from experience?

In the file named "check_consistency.cu".
If we have left image with black pixel(0), we are forcing the corresponding disparity value of the pixel value to 0.

Why its necessary to do?

Thanks,

Why is it incorrect when it comes to SUBPixel Mode? When the SUBPIXEL Parameter was set to true,the result disparity image value is incorrect .Did I miss something?

While it is easy to obtain the disparity for left image using ssgm.execute(left.data, right.data, leftDisparity.data);, if I want to obtain the disparity for right image by changing it to ssgm.execute(right.data, left.data, rightDisparity.data); it gives me garbage disparity.

What is the correct way of obtaining disparity for right image? Thanks so much!

I am trying to build the code on my system:

Ubuntu 14.04 LTS
Nvidia Titan X (Maxwell architecture, sm_52)
Cuda 8.0

I have also manually compiled and installed GLFW3 as explained here. Here is the result of cmake ..

cmake_result.txt

When I make the project it fails because of some missing openCV functions that are used for target stereo_test.

You can see the error here

make_result.txt

I installed OpenCV from terminal. If I remember correctly it is OpenCV 2.7

What is the problem with my openCV installation? Should I upgrade to CV3 or it should work also with CV2?

#5

I have managed to run the code on some stereo pair images from the Middlebury dataset and the result is quite ok.

But when I try to use it on other rectified stereo pairs from my own cameras or from other datasets the result is much different (and much worse).

Could you please provide references to the papers on which the code is based? That would help in understanding the code.

Can you also give some hints on how to optimize it to run on different stereo datasets/images? I saw right now only one parameter can be passed, the max_disparity, and it can be only 64 or 128. Why is it so?

Are there any other parameters that can be set inside the code? I am thinking about the classic parameters of SGM: window_size, cost penalties p1 and p2, number of search lines for semi-global-matching (4,8,16...) et cetera!

Thanks for your help!

I get error when executing
~/libSGM/build/sample/image$ ./stereo_test image_left.png image_right.png. The images are in folder /libSGM/sample/image/.
It throws error input image format must be CV_8U or CV_16U Wonder does it support other image format other than .pgm? I am on Ubuntu 16.04, Cmake 3.5.1, Opencv 3.4.0, Python 3.5. Thanks in advance!

My os is windwos10 64, GTX1060, I got error like "nvcc fatal : Host compiler targets unsupported OS"

I have the following configurations.

Win32 x86
CUDA 6.5
Visual Studio 12 2013
GeForce 730 with Compute Capability 3.5
OpenCV 3.4.3

I have tried building libSGM but getting errors (like __shfl_down,__shfl_xor,__shfl_up not defined) in horizontal_path_aggregation.cu

Will libSGM build for my configuration ? If so, how do I fix it ?

Hello,i have understood the algorithm .Can you provide some papers about the CUDA implement of the algorithm .I have tried implement it by CUDA,but it is slower than your implement. So i prefer you to provide some papers about it.I will appreciate it very much.Thank you very much.

Hi, I test the program correctly in PC using cmake, and I want to debug cuda kernal function with parameters "-G". However, some bugs are here, like median_filter: unkown error.

This mistake confused me for a long time. Looking forward to your reply.
Thanks and Regards,
John

Should i run it again with exchanged left and right image,if i want get the disparity of right image or i just use the right disparity which is generated by left base image? I think the both method are different.

I tried to run different StereoSGM instances in parallel in ROS, so each can compute disparity for different sizes of image pairs. However, it is giving me weird disparity values (shown below).

On the other hand, when I run a simple test in the sample, everything works just fine (see below). My images are just an empty filed with an object on the left.

It seems to me that it is not safe to initialize multiple instance of StereoSGM and run them in parallel. Is this the case? How can we have the library support ROS?

Any idea on why am I seeing below disparity in ROS? I've checked the data type for disparity (16SC1 normalized to 8UC1), version of OpenCV is the same (3.3.1 for ROS Kinetic). The images below was not even generated on ROS messages. They are actually exact duplication and the image pairs were simply being imread from file.

Update 08/29
I magically got ROS working. I don't remember what things I changed. I run the program overnight and it didn't crash at all, and the disparity quality is very good. I'll test again and see if the problem occurs again, or it could just be bugs in my code that I didn't catch...

Good disparity in sample test:

Bad disparity in ROS:

Linking CXX executable stereo_movie
/usr/local/bin/ld: /usr/local/lib/libglfw3.a(x11_window.c.o): undefined reference to symbol 'XConvertSelection'
/usr/lib/gcc/x86_64-linux-gnu/4.6/../../../x86_64-linux-gnu/libX11.so: error adding symbols: DSO missing from command line
collect2: ld returned 1 exit status
make[2]: *** [sample/movie/stereo_movie] 错误 1
make[1]: *** [sample/movie/CMakeFiles/stereo_movie.dir/all] 错误 2
make: *** [all] 错误 2

Hi guys. Thanks for the awesome library. It has been working great.

On regions such as the sky, the disparity can be very noisy. My questions are

1. Are there any parameters that we can tweak to get more robust detection on sky where there is virtually no pattern? Ideally we want to output disparity values of 0 on sky.
2. Are there any ways to get the confidence map for disparity so that we can post filter ourselves?

Thanks!

The Readme says that

$ cd sample/movie/
$ ./stereo_movie left_image_%04d.pgm right_image_%04d.pgm

to execute sample. but there is no stereo_movie file? Instead a stereosgm_movie .cpp.
So How should I do?
Sorry, I'm new to cmake. I can't figure out how to make them function.

Is there a way I can feed a GPU mat directly into the SGM function? Right now I have to download my GPU mat to a CPU mat then push it through the SGM function so it can (presumably) re-upload it to the GPU. Seems kinda inefficient.

hello
Can I build this project with cuda 5.5 ?
when I built it, error of "sm_50" and "identifier "__shfl" is undefined" occured.
what do I if I want to run it with cuda 5.5 ?

I'd like to add a refinement step of the disparities after the winner_takes_all step. Therefore, I'd need to know at which line of code the acutual matching costs per pixel are obtained.

The winner_takes_all kernel receives the smoothed costs (d_scost) and somewhere at the end of the kernel the disparity is calculated for left (leftDisp) and right (rightDisp). I'm somewhat confused where the actual matching costs are obtained. I'm guessing it could be lhv or minCostL2.

I do not understand what if the min disparity of my input data is higher than 128.According to the code , it won't get correct disparity image.

Hello！ Recently, I have been studying Semi-Global Matching of NVIDIA VisionWorks https://developer.nvidia.com/embedded/visionworks . And I compared the running time of this code with your code. I found that the runnning time of the step winner_takes_all.cu is twice the runtime of corresponding step nvxComputeDisparityNode() in Semi-Global Matching code. I'm thinking about how to improve the ‘winner_takes_all.cu’ code. Do you have some improvement plans or suggestions? Thank you.

How to run this code in the windows

libSGM/src/check_consistency.cu : 52 invalid device function
I have given an image of size 1200x320 which is divisible by 16 ,yet I am getting the above error.Any suggestion is appreciable.

Hi,
I am able to compile and run libSGM,the thing is that when I run the executable the output is different for two different runs with the same input images.I am writing the disparity into a bin file and verifying it in the matlab.
I am using the standard samples provided by the developers.Please help solve the issue.

Hi,

• Environment - TX2
• Running - L4T 16.04

I was using your beautiful SGM implementation in CUDA and noted the following things.

I was looking to up the optimize the performance as my image resolution is big and FPS was too low. So check if there was room for any improvement right from census transform till the consistency check. I must say it is pretty tight and most of them are already optimized.

But I found a weird behaviour in all the path aggregation kernels. I was using 4 path, So there are 2 horizontal and 2 vertical kernel calls. Each kernel call took around 6-7ms which is good. Profiler also asid all the 4 kernels took 25ms(attached a screenshot of the profiler) concurrently but when I used gettimeofday() to measure the time for the entire path_aggregation process, the result was surprisingly around 45ms+. There was a huge 20ms difference.

In sgm.cu :

gettimeofday(&start, NULL);

m_path_aggregation.enqueue(
			m_census_left.get_output(),
			m_census_right.get_output(),
			width, height,
			penalty1, penalty2,
			stream);

cudaDeviceSynchronize();
gettimeofday(&stop, NULL);
secs = (double)(stop.tv_usec - start.tv_usec) / 1000000 + (double)(stop.tv_sec - start.tv_sec);
printf("--------------PATH AGGREGATION  -> TIME IN MILLISECONDS : %f\n",secs * 1000);

The print here gave 45ms+ where as the profiler suggests all the kernels(2 vertical + 2 horizontal) are executed in 25ms. Is there any other places where the time is being occupied? I don't think there mem copies take too much time as well. And I would be happy to hear it from you about where else the code can be optimized speed-wise? Any leads would be appreciated.

Hello,

I am interested in increasing the disparity search to 256.
I see that scan_cost and winner_takes_all are the two function that would probably need modification to get higher disparity size working.
But the way 64 and 128 disparity search are implemented it seems 256 would not be straight forward extension to 128 version.
Any suggestion or pointer to how to do it?

Thanks,

Hello, I try to use your library on JetsonTX2, but it fails with error message:

OpenCV Error: Gpu API call (unknown error) in allocate, file /home/nvidia/workspace/opencv/modules/core/src/cuda/gpu_mat.cu, line 116
/home/nvidia/workspace/opencv/modules/core/src/cuda/gpu_mat.cu:116: error: (-217) unknown error in function allocate

The size of the frame is 128 x 96, so I don't think that the problem is out of the memory in global memory.

Did you see this problem early on some devices?

Best wishes.