asdlei99 / sarpn

Structure-Aware Residual Pyramid Network for Monocular Depth Estimation IJCAI 2019

Structure-Aware Residual Pyramid Network for Monocular Depth Estimation

This is the implementation of the paper Structure-Aware Residual Pyramid Network for Monocular Depth Estimation, IJCAI 2019, Xiaotian Chen, Xuejin Chen, and Zheng-Jun Zha.

Citation

@inproceedings{Chen2019structure-aware,
             title = {Structure-Aware Residual Pyramid Network for Monocular Depth Estimation},
             author = {Chen, Xiaotian and Chen , Xuejin and Zha, Zheng-Jun},
	     conference={International Joint Conferences on Artificial Intelligence},
             year = {2019}   
} 

Contents

1. Introduction
   
2. Usage
   
3. Results
   
4. Acknowledgements
   

Introduction

Monocular depth estimation is an essential task for scene understanding. The underlying structure of objects and stuff in a complex scene is critical to recovering accurate and visually-pleasing depth maps. Global structure conveys scene layouts,while local structure reflects shape details. Recently developed approaches based on convolutional neural networks (CNNs) significantly improve the performance of depth estimation. However, few of them take into account multi-scale structures in complex scenes. In this paper, we propose a Structure-Aware Residual Pyramid Network (SARPN) to exploit multi-scale structures for accurate depth prediction. We propose a Residual Pyramid Decoder (RPD) which expresses global scene structure in upper levels to represent layouts, and local structure in lower levels to present shape details. At each level, we propose Residual Refinement Modules (RRM) that predict residual maps to progressively add finer structures on the coarser structure predicted at the upper level. In order to fully exploit multi-scale image features, an Adaptive Dense Feature Fusion (ADFF) module, which adaptively fuses effective features from all scales for inferring structures of each scale, is introduced.

Usage

Dependencies

• Python3.6
• PyTorch(1.0.1)
• NYU Depth v2

Train

As an example, use the following command to train SARPN on NYUDV2.

CUDA_VISIBLE_DEVICES="0,1,2,3" python train.py --trainlist_path (the path of trainlist(nyu2_train.csv))\
					       --checkpoint_dir (the directory to save the checkpoints)\
					       --root_path (the root path of dataset)\
					       --logdir (the directory to save logs and checkpoints)\
					       --pretrained_dir (the path of pretrained models)\
					       --do_summary

Evaluation

Use the following command to evaluate the trained SARPN on NYUDV2 test data.

CUDA_VISIBLE_DEVICES="0" python evaluate.py --testlist_path (the path of testlist(nyu2_test.csv))\
					    --root_path (the root path of dataset)\
					    --loadckpt (the path of the loaded model)\
					    --pretrained_dir (the path of pretrained models)\
					    --threshold (threshold of the pixels on edges)

Pretrained Model

You can download the pretrained model:
NYUDV2

Pre-processed Data

You can download the pre-processed data from this link, which is shared by Junjie Hu.

Results

Acknowledgements

Thanks to Junjie Hu for opening source of his excellent work. Our work is inspired by this work and part of codes.