mjitg / pytorch-hitnet-hierarchical-iterative-tile-refinement-network-for-real-time-stereo-matching

HITNet implementation using PyTorch

PyTorch-HITNet-Hierarchical-Iterative-Tile-Refinement-Network-for-Real-time-Stereo-Matching

HITNet implementation using PyTorch

This is a repository including code implementing Google paper HITNet: Hierarchical Iterative Tile Refinement Network for Real-time Stereo Matching

This project is an initial version, which can train and test the model but may contain some errors and need further modification and debugging. If you find any issue about my code, please open issues or contact me ([email protected]) as soon as possible.

Currently this project cannot reproduce the accuracy and speed reported in the original paper. In terms of the speed, the official implementation uses their optimized cuda op to accelerate the reference and training.(Please refer to their official repository, which has not inculded the model code yet).

Thanks for the help of Vladimir Tankovich, who has proposed this great stereo network with his team and provided me with a lot of details and clarifications of the original paper.

Also, I would thank @xy-guo, who proposed the amazing GwcNet, since the code was partially borrowed from his repository.

Requirements

Pytorch = 1.1 Python = 3.6.8 CUDA10

Slant Ground-truth

Slant parameter GT is here, which is generated using least square fit and RANSAC. The code for slant parameter generation is here.

pytorch-hitnet-hierarchical-iterative-tile-refinement-network-for-real-time-stereo-matching's Issues

question in class DispUpsampleBySlantedPlane

hi,
Can you help me explain what ts means in the class DispUpsampleBySlantedPlane ?

"maxdisp" in initialization_loss.py

there's a "maxdisp" in "subpix_cost" function in "loss/initialization_loss.py", line 28 to be accurate. But it seems to be a bug? It hasn't been used as a hyperparameter, nor a passed temporal parameter.

Cuda error: device-side assert triggered

When it runs training, it always shown this error after some iterations:

/opt/conda/conda-bld/pytorch_1565287025495/work/aten/src/THC/THCTensorScatterGather.cu:100: void THCudaTensor_gatherKernel(TensorInfo<Real, IndexType>, TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [0,0,0], thread: [6,0,0] Assertion indexValue >= 0 && indexValue < src.sizes[dim] failed.
Traceback (most recent call last):
File "/home/workspace/HITNet/PyTorch-HITNet/main.py", line 297, in
train()
File "/home/workspace/HITNet/PyTorch-HITNet/main.py", line 125, in train
loss, scalar_outputs, image_outputs = train_sample(sample, compute_metrics=do_summary)
File "/home/workspace/HITNet/PyTorch-HITNet/main.py", line 225, in train_sample
disp_gt, dx_gt, dy_gt, args.maxdisp)
File "/home/workspace/HITNet/PyTorch-HITNet/loss/total_loss.py", line 46, in global_loss
lambda_init * init_loss(cv, d_gt_pyramid[i], maxdisp)[mask]
File "/home/workspace/HITNet/PyTorch-HITNet/loss/initialization_loss.py", line 16, in init_loss
cost_nm = torch.gather(pred_init_cost, 1, get_non_match_disp(pred_init_cost, d_gt))
File "/home/workspace/HITNet/PyTorch-HITNet/loss/initialization_loss.py", line 50, in get_non_match_disp
INF = torch.Tensor([float("Inf")]).view(1, 1, 1, 1).repeat(B, D, H, W).to(d_gt.device)
RuntimeError: CUDA error: device-side assert triggered

Have you ever met this before? Or any ways to solve to this?

The code for slant parameter generation is Require access rights

Thanks for your share slant generation code, but access denied when I access the code link, I have clicked the permission request, Have you received the permission request, Thank you very much~

request for video demonstration of paper

Hello Mjitg, I see that you have implemented the code according to the paper by making different submodules, but because we don't have any video demonstration of how the architecture is working we are unable to understand it, so it would be great if you could upload a video demonstrating the algorithm, it need not cover all the part, could just explain the model and loss function, that will be more than enough.

Thank you

Akanksh

how to generate Slant parameter GT

Thanks for your work， This is a very perfect work,
I want to used myself dataset for your code, but i want to konw how to generate Slant parameter GT, Can you share Detailed generation process, Thank you very much!!!

Data inconsistency

Hi.
There are 2 .pfm images not existing according to the slant data you provided:
it seems 000114_10.pfm & 000115_10.pfm are missing?

I'm having trouble in inference

After Kitti trained the network, the model was much less effective in inference. But it's not an overfit. Because I'm using the test set to inference.and The dataload for test and inference is the same. Does anyone have a problem with that.

Great work！ The performance details

Great work！ Could you show the performance of your code in terms of EPE and >npixl metric ? How close are they to the original paper?

Pretrained models

Thanks a lot for your work! Are the trained models available and can you show the quantitative results to compare your implementation with the original one? Thanks ahead!