leahlyu22 / outdoor-3d-scene-reconstruction

Combine URF with neural-scene-graphs for dynamic scenes

Outdoor 3d Scene Reconstruction

Project description

The project aimed to improve the performance of outdoor scene reconstruction, with a specific focus on the sky regions by proposing and implementing a novel algorithm.

Urban Radiance Field (Google Research) is used as the theoretical basis combining with Neural Scene Graphs for Dynamic Scene (CVPR 2021) for this project. Original repository forked from the Implementation of Original Neural Scene Graph Implementation, original readme.

To enhance the supervisory signal in sky regions, an additional MLP model is added to the sky region with a pretrained segmentation DeepLabV3Plus model.

---

Model performance

The final loss is 0.38 compared to the starting point of 0.79.
The final psnr is 22.13 compared to the starting point of 10.26.

The top right image is the rendering result only for non-sky-region. The middle right image is the result of combining both sky and the non-sky region. The bottom right image is the ground-truth image.

The result of sky-region reconstruction:

Check the video result here:

---

Citation

@InProceedings{Ost_2021_CVPR,
    author    = {Ost, Julian and Mannan, Fahim and Thuerey, Nils and Knodt, Julian and Heide, Felix},
    title     = {Neural Scene Graphs for Dynamic Scenes},
    booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
    month     = {June},
    year      = {2021},
    pages     = {2856-2865}
}

@article{rematas2022urf,
    title={Urban Radiance Fields},
    author={Konstantinos Rematas and Andrew Liu and Pratul P. Srinivasan and Jonathan T. Barron and Andrea Tagliasacchi and Tom Funkhouser and Vittorio Ferrari},
    journal={CVPR},
    year={2022}
}

outdoor-3d-scene-reconstruction's Issues

Open source ?

Hi, you do very good job!
Will you open source of this project ?