Neural Surface Reconstruction of Dynamic Scenes with Monocular RGB-D Camera
Hongrui Cai · Wanquan Feng · Xuetao Feng · Yan Wang · Juyong Zhang
NeurIPS 2022
Paper | Project Page | Poster
We propose Neural-DynamicReconstruction (NDR), a template-free method to recover high-fidelity geometry, motions and appearance of a dynamic scene from a monocular RGB-D camera.
Usage
Data Convention
The data is organized as NeuS
<case_name>
|-- cameras_sphere.npz # camera parameters
|-- depth
|-- # target depth for each view
...
|-- image
|-- # target RGB each view
...
|-- mask
|-- # target mask each view (For unmasked setting, set all pixels as 255)
...
Here cameras_sphere.npz
follows the data format in IDR, where world_mat_xx
denotes the world-to-image projection matrix, and scale_mat_xx
denotes the normalization matrix.
Pre-processed Data
You can download a part of pre-processed KillingFusion data here and unzip it into ./
.
Important Tips: If the pre-processed data is useful, please cite the related paper(s) and strictly abide by related open-source license(s).
Setup
Clone this repository and create the environment (please notice CUDA version)
git clone https://github.com/USTC3DV/NDR-code.git
cd NDR-code
conda env create -f environment.yml
conda activate ndr
Dependencies (click to expand)
- torch==1.8.0
- opencv_python==4.5.2.52
- trimesh==3.9.8
- numpy==1.21.2
- scipy==1.7.0
- PyMCubes==0.1.2
Running
- Training
python train_eval.py
- Evaluating pre-trained model
python pretrained_eval.py
Data Pre-processing
Coming Soon
Geometric Projection
- Compiling renderer
cd renderer && bash build.sh && cd ..
- Rendering meshes
Input path of original data, path of results, and iteration number, e.g.
python geo_render.py ./datasets/kfusion_frog/ ./exp/kfusion_frog/result/ 120000
The rendered results will be put in dir [path_of_results]/validations_geo/
Todo List
- More Pre-processed Data
- Code of Data Pre-processing
- Code of Geometric Projection
- Pre-trained Models and Evaluation Code
- Training Code
Acknowledgements
This project is built upon NeuS. Some code snippets are also borrowed from IDR and NeRF-pytorch. The pre-processing code for camera pose initialization is borrowed from Fast-Robust-ICP. The evaluation code for geometry rendering is borrowed from StereoPIFu_Code. Thanks for these great projects. We thank all the authors for their great work and repos.
Contact
If you have questions, please contact Hongrui Cai.
Citation
If you find our code or paper useful, please cite
@inproceedings{Cai2022NDR,
author = {Hongrui Cai and Wanquan Feng and Xuetao Feng and Yan Wang and Juyong Zhang},
title = {Neural Surface Reconstruction of Dynamic Scenes with Monocular RGB-D Camera},
booktitle = {Thirty-sixth Conference on Neural Information Processing Systems (NeurIPS)},
year = {2022}
}
If you find our pre-processed data useful, please cite the related paper(s) and strictly abide by related open-source license(s).