haram-kim / hetero_stereo_matching

Hetero_Stereo_Matching

Hetero Stereo Matching

The source code is released under the MIT License.

Project Page

You can see the high-quality video at:
https://haram-kim.github.io/Hetero_Stereo_Matching/

Prerequisites

Python

We tested our code on Python version 3.8.0
[Download]

OpenCV

We use OpenCV to visualize and manipulate images.

CUDA

We use CUDA for parallel computation.
[Download] [Installation guide]

pyCUDA

We use python wrapper pyCUDA for CUDA.

Others

You can install dependent libararies by running :

pip install opencv pycuda pyyaml scipy tqdm h5py hdf5plugin

Installation & Running guide

The code works best on Ubuntu 20.04

0. Download DSEC data set
   DSEC [Download]

   0-1. (Optional) If you want to download only the preprocessed data, please download:
   'interlaken_00_c_processed_data' (8.71GB)
   'interlaken_00_d_processed_data' (34.4GB)
   'interlaken_00_e_processed_data' (26.9GB)
   'interlaken_00_f_processed_data' (14.6GB)
   'interlaken_00_g_processed_data' (14.2GB)
   (Still, '[DATA_SEQUENCE]_calibration' must be download.)

Directory structure :

/PATH_TO_DSEC
├────interlaken_00_c
│    ├────interlaken_00_c_calibration
│    ├────interlaken_00_c_disparity_event
│    ├────interlaken_00_c_disparity_image
│    ├────interlaken_00_c_events_left
│    ├────interlaken_00_c_events_right
│    ├────interlaken_00_c_images_rectified_left
│    ├────interlaken_00_c_images_rectified_right
│    ├────interlaken_00_c_disparity_timestamps.txt
│    └────interlaken_00_c_image_timestamps.txt
│
├────interlaken_00_c_processed_data.hdf5 (Please locate the preprocessed data to this.)

...


├────interlaken_00_g
│    ├────interlaken_00_g_calibration
│    
│    ...
│    
│    └────interlaken_00_g_image_timestamps.txt
└────interlaken_00_g_processed_data.hdf5

1. Clone this repository:

$ git clone https://github.com/Haram-kim/Hetero_Stereo_Matching.git

2. Run the code

$ cd HSM
$ python main.py [PATH_TO_DSEC] [DATA_SEQUENCE]
Example)  $ python main.py /c/DSEC/ interlaken_00_c

Configuration settings

config/config.yaml

Feature tracker

feature num - the number of features
track_err_thres - KLT traker error threshold
track_win_size - KLT tracker window size
extract_win_size - Feature extractor window size

feature num: 1000  
track_err_thres: 10
track_win_size:
- 21
- 21
extract_win_size: 12

Disparity estimator

disparity_range - disparity range (0 - max_disp)
kernel_radius - patch radius
ncc_gaussian_std - standard deviation of Gaussian filter
msd_gap - Maximum Shift Distance interval

disparity_range: 100
kernel_radius: 12
ncc_gaussian_std: 2
msd_gap: 10

Time estimation mode

If you want to estimate the computation time, you can run the time_estimation mode.

time_estimation: True

Visualize

'show disparity inlier' requires ground truth disparity.
If you want to see all estimated disparity of the proposed method, please change the flag 'show_disparity' False to True
If you want to see estimated disparity on edges of the proposed method, please change the flag 'semi_dense' False to True

show_disparity_inlier: True
show_disparity: True
semi_dense: True

hetero_stereo_matching's Issues

Should metrics be calculated on densified disparity GT?

I was reading the code, and I noticed that the metrics (RMSE, MAE, ...) are calculated against densified disparity GT (the original GT convolved with a gaussian kernel), instead of the original GT.
I ran the evaluation code after changing the metric GT to the original GT. The performance becomes slightly lower than the reported values.

Results with other datasets

Have you tested this method with other datasets?
Does your stereo matching method work if you have objects moving in the scene at different depths?

pycuda error

Hi,

It's a great job!

However, I meet the above error when trying to test HSM on an Nvidia 3090 GPU:

I find that lowering the self.cuda_SBS = (32, 16, 1) works, but the resulting disparity seems unsatisfactory:
self.cuda_SBS = (24, 16, 1):

self.cuda_SBS = (16, 16, 1):

What should I do to get satisfactory results?

The RMSE and MAE are only calculated on inliers

In the function "metric", "disparity" only has inliers. "disparity_all" is the actual results.

Hetero_Stereo_Matching/HSM/HSM.py

Lines 382 to 412 in cc3389e

	def metric(self, method, disparity, disparity_gt, err_thres):
	depth = self.f_b/(disparity + 1e-9)
	depth_gt = self.f_b/(disparity_gt + 1e-9)
	inlier_idx = np.where((np.abs(disparity_gt - disparity) < err_thres) * (disparity_gt > 0))
	all_idx = np.where((disparity_gt > 0) * (disparity_gt > 0))
	gt_idx = np.where(disparity_gt > 0)
	if not method in self.eval_data.keys():
	self.eval_data[method] = dict()
	if len(self.eval_data[method].keys()) == 0:
	self.eval_data[method]['disparity'] = disparity[inlier_idx]
	self.eval_data[method]['disparity_all'] = disparity[all_idx]
	self.eval_data[method]['depth'] = depth[inlier_idx]
	self.eval_data[method]['gt'] = dict()
	else:
	self.eval_data[method]['disparity'] = np.append(self.eval_data[method]['disparity'], disparity[inlier_idx])
	self.eval_data[method]['disparity_all'] = np.append(self.eval_data[method]['disparity_all'], disparity[all_idx])
	self.eval_data[method]['depth'] = np.append(self.eval_data[method]['depth'], depth[inlier_idx])
	if len(self.eval_data[method]['gt'].keys()) == 0:
	self.eval_data[method]['gt']['disparity'] = disparity_gt[inlier_idx]
	self.eval_data[method]['gt']['disparity_all'] = disparity_gt[all_idx]
	self.eval_data[method]['gt']['depth'] = depth_gt[inlier_idx]
	self.eval_data[method]['gt']['depth_valid_num'] = depth_gt[gt_idx].shape[0]
	else:
	self.eval_data[method]['gt']['disparity'] = np.append(self.eval_data[method]['gt']['disparity'], disparity_gt[inlier_idx])
	self.eval_data[method]['gt']['disparity_all'] = np.append(self.eval_data[method]['gt']['disparity_all'], disparity_gt[all_idx])
	self.eval_data[method]['gt']['depth'] = np.append(self.eval_data[method]['gt']['depth'], depth_gt[inlier_idx])
	self.eval_data[method]['gt']['depth_valid_num'] += depth_gt[gt_idx].shape[0]

However, when you calculate RMSE and MAE in "metric_print", you use "disparity" instead of "disparity_all".

Hetero_Stereo_Matching/HSM/HSM.py

Lines 414 to 431 in cc3389e

	def metric_print(self, method):
	try:
	d_data = self.eval_data[method]['disparity']
	d_data_all = self.eval_data[method]['disparity_all']
	z_data = self.eval_data[method]['depth']
	d_gt = self.eval_data[method]['gt']['disparity']
	d_gt_all = self.eval_data[method]['gt']['disparity_all']
	z_gt = self.eval_data[method]['gt']['depth']
	valid_gt = self.eval_data[method]['gt']['depth_valid_num']
	error = dict()
	error['RMSE'] = RMSE(d_data, d_gt)
	error['MAE'] = MAE(d_data, d_gt)
	error['DTdelta'] = DTdelta(d_data, d_gt, valid_gt)
	error['RTdelta'] = RTdelta(d_data_all, d_gt_all)
	error['depth_RMSE'] = RMSE(z_data, z_gt)
	error['depth_ARD'] = ARD(z_data, z_gt)
	error['depth_ATdelta'] = ATdelta(z_data, z_gt, valid_gt)

This makes the corresponding result numbers in the paper very misleading.