jingsenzhu / indoorinverserendering

[SIGGRAPH Asia'22] Learning-based Inverse Rendering of Complex Indoor Scenes with Differentiable Monte Carlo Raytracing

News

• 04/12/2022 repository created
• 31/12/2022 code release for material-geometry network
• 17/01/2023 dataset release: InteriorVerse material-geometry part
• 24/02/2023 code release for lighting network
• 28/02/2023 pretrained model and testing data (object insertion) released

TODO

• Code release for Material-Geometry network
• Code release for Lighting network
• Release of pretrained model
• Dataset release: InteriorVerse material-geometry part
• Dataset release: InteriorVerse lighting part

Learning-based Inverse Rendering of Complex Indoor Scenes with Differentiable Monte Carlo Raytracing

Project Page | Paper | Dataset

This repository implements the paper "Learning-Based Inverse Rendering of Complex Indoor Scenes with Differentiable Monte Carlo Raytracing" in SIGGRAPH Asia'22. It includes training and testing code of material-geometry network (MGNet) and testing code of lighting network (LightNet).

Also check our following work: I²-SDF !

Pretrained Models

Pretrained models are available here, including MGNet and LightNet.

Citation

If you find our work is useful, please consider cite:

@inproceedings{zhu2022learning,
    author = {Zhu, Jingsen and Luan, Fujun and Huo, Yuchi and Lin, Zihao and Zhong, Zhihua and Xi, Dianbing and Wang, Rui and Bao, Hujun and Zheng, Jiaxiang and Tang, Rui},
    title = {Learning-Based Inverse Rendering of Complex Indoor Scenes with Differentiable Monte Carlo Raytracing},
    year = {2022},
    publisher = {ACM},
    url = {https://doi.org/10.1145/3550469.3555407},
    booktitle = {SIGGRAPH Asia 2022 Conference Papers},
    articleno = {6},
    numpages = {8}
}

indoorinverserendering's Issues

Will scenes be available in 3D formats, such as obj, fbx, glb, etc.?

Hello. Thank you very much for the wonderful paper!
Will scenes be available in 3D formats, such as obj, fbx, glb, etc.?

The download permission of the complete data set

I have applied for the download permission of the complete data set and sent you an email. Could you please check it? My email is [email protected],Thanks !

training of lightnet, lighting dataset release.

Hi. Thanks for sharing your wonderful research. Do you not share the training part of lightNet? I want to do a comparison on different datasets.

And when will the InteriorVerse Lighting dataset be released?

thank you

Dataset

When do you plan to release the dataset?

Unable to correctly load exr format

Hello,

I am unable to load the images using your provided code. The best I get using imageio results in a very dark image. Can you help with me this pls?

GPU memory usage

Hi there, during the ray tracing process, when you consider multiple light rays for each pixels, let's say S, each ray requires multiple samples from NeRF, let's say N. For each image with size H times W, the input batch size of NeRF is S*N*H*W. When (S, N, H, W)=(32, 64, 640, 480), the input batch size is nearly 6.3*10^8, which is a huge cost of memory during NeRF's forward pass, since NeRF stores the partial derivatives of input to its weights. The storage cost is around 30~40GB, which is usually much larger than a GPU's RAM. May I ask how do you solve the problem? Thank you.

Spatially-varying lighting dataset release

Hello, thank you for this great paper. When will you release the spatially-varying lighting dataset?

Access to InteriorVerse Dataset

Hi, I am a undergraduate student from National Taiwan University. We are looking for high quality inverse rendering datasets for our research. May I have an access to your dataset?

Full release of code and data?

Hi there, great work with the paper and the dataset! I wonder if there is any plan for full release of the pretrained models and the code including the lighting part?

During training--------NaN or Inf found in input tensor.