jingsenzhu / i2-sdf

[CVPR 2023] I^2-SDF: Intrinsic Indoor Scene Reconstruction and Editing via Raytracing in Neural SDFs

News

• 04/04/2023 dataset preview release: 2 synthetic scenes available
• 15/04/2023 code release: 3D reconstruction and novel view synthesis part
• 21/04/2023 dataset release: real data

TODO

• Full dataset release
• Code release for 3D reconstruction and novel view synthesis
• Code release for intrinsic decomposition and scene editing

Dataset released

• Synthetic: kitchen_0, bedroom_relight_0, bedroom_0, bedroom_1, bedroom_relight_1, diningroom_0, livingroom_0, livingroom_1, more scenes to be released
• Real: inria_livingroom, nisr_livingroom, nisr_coffee_shop_0, nisr_coffee_shop_1, release complete

I²-SDF: Intrinsic Indoor Scene Reconstruction and Editing via Raytracing in Neural SDFs (CVPR 2023)

Project Page | Paper | Dataset

Setup

Installation

conda env create -f environment.yml
conda activate i2sdf

Data preparation

Download our synthetic dataset and extract them into data/synthetic. If you want to run on your customized dataset, we provide a brief introduction to our data convention here.

Dataset

We provide a high-quality synthetic indoor scene multi-view dataset, with ground truth camera pose and geometry annotations. See HERE for data conventions. Click HERE to download.

3D Reconstruction and Novel View Synthesis

Training

python main_recon.py --conf config/<config_file>.yml --scan_id <scan_id> -d <gpu_id> -v <version>

Note: config/synthetic.yml doesn't contain light mask network, while config/synthetic_light_mask.yml contains.

If you run out of GPU memory, try to reduce the split_n_pixels (i.e. validation batch size), batch_size in the config. The default parameters are evaluated under RTX A6000 (48GB). For RTX 3090 (24GB), try to set split_n_pixels 5000.

Evaluation

Novel view synthesis

python main_recon.py --conf config/<config_file>.yml --scan_id <scan_id> -d <gpu_id> -v <version> --test [--is_val] [--full]

The optional flag --is_val evaluates on the validation set instead of training set, --full produces full-resolution rendered images without downsampling.

View Interpolation

python main_recon.py --conf config/<config_file>.yml --scan_id <scan_id> -d <gpu_id> -v <version> --test --test_mode interpolate --inter_id <view_id_0> <view_id_1> [--full]

Generates a view interpolation video between 2 views. Requires ffmpeg being installed.

The number of frames and frame rate of the video can be specified by options.

Mesh Extraction

python main_recon.py --conf config/<config_file>.yml --scan_id <scan_id> -d <gpu_id> -v <version> --test --test_mode mesh

Intrinsic Decomposition and Scene Editing

Brewing🍺, code coming soon.

Citation

If you find our work is useful, please consider cite:

@inproceedings{zhu2023i2sdf,
    title = {I$^2$-SDF: Intrinsic Indoor Scene Reconstruction and Editing via Raytracing in Neural SDFs},
    author = {Jingsen Zhu and Yuchi Huo and Qi Ye and Fujun Luan and Jifan Li and Dianbing Xi and Lisha Wang and Rui Tang and Wei Hua and Hujun Bao and Rui Wang},
    booktitle = {CVPR},
    year = {2023}
}

Acknowledgement

• This repository is built upon Pytorch lightning.
• Thanks to Lior Yariv for her excellent work VolSDF.
• Thanks to Scalable-NISR team for providing their real-world dataset.

i2-sdf's Issues

Some weired floater in the extracted mesh

Thanks for sharing the great work！
But when I ran the mesh extract for scan 31 and scan 127 I got very weired look, like the following:

The main part of the scan looks great, but there still some floater or redundant plane. Do you have any idea about this?
The environment information is:
tinycudann 1.7
torch 1.13.1+cu117
Ubuntu 22.04.2 LTS
NVIDIA 4090
Driver Version: 525.116.04

Material labels for released dataset?

Hi there, thanks for releasing the two demo scenes! I wonder if there is a plan to include materials and emitter mask in the current/later version (as indicated here), which will be useful for inverse rendering tasks.

Thanks!

conf.plot.grid_boundary not defined

conf.plot.grid_boundary (code here) is not defined in any of the provided config files. Shall I set it to grid_boundary = [-1.1, 1.1]?

Missing config file

Hi,
Great work and thanks for sharing! I wanted to test your code after your last update, however, I can't find an example of a config file or documentation for it. Is this something you could provide?

When the code will be released?

Hello. Thank you for sharing this amazing project!

May I know when the code will be released?

I am looking forward to running this project!

Hi, when will the code be released?

Hi, when will the code be released?

Dataset camera intrinsics

Hi,

Thanks for sharing this work. Does the dataset contain any camera intinsics information, such as focal length / fov, optical center, etc? It seems like cameras.npz contains only the camera to world transformation matrix.

Thanks!

Pretrained weights

Any possibility to provide pretrained model or weights

Code release for scene editing

Thanks for the paper!
When are you planning on releasing the code for scene editing?

what's depth/normal supervision ?

Hi,

Great work. I have questions regarding equation 9) in the paper.
What are the depth/normal supervision ? specifically, what are D(r) and N(r) in 11) and 12) respectively.

1. monosdf uses public models to generate depth maps and normal maps to supervise the model. Just curious how to generate depth/normal supervision for i2sdf.
2. Do i2sdf and monosdf share the same depth/normal supervision ?

Thanks

Artifacts with depth maps?

Hi there I noticed some potential issues with a few depth maps, where pixels that belong to a open window have large depth values, instead of 0. Here is the 0003 image of the released bedroom scene.

Note the yellow strip in the center figure, which are inside the white area of the 'window hole'; the pixels have large depth values but ideally should be infinite (0) because they correspond to the outdoor env. I pick two locations (red dot and green dot) on the figures, and query the depths, getting 4.3840003 3.1920002 respectively, but the former one should be 0.

As a result, if you simply back-project depth to 3D, you get fantom geometry (in red circle).

Not sure if this is an issue with the way I read the depth file (cv2.imread(filename, -1)), or the depth file itself.

Inconsistent normal losses

I wanted to let you know about an inconsistency in the code.

The code seems to contain two normal losses:

1. get_normal_l1_loss, which actually computes an angular loss, as pointed out in Equation 12 of your paper. This name seems confusing, as it does not compute the L1 normal loss.
2. get_normal_angular_loss, which actually computes the truncated scaled angle.

However, the loss computation in forward uses loss 1 above twice (see this line), weighted by (self.normal_weight + self.angular_weight), and loss 2 above is not used at all.

Code release for intrinsic decomposition

Thank you for your excellent work, I'm wondering when will the code for intrinsic decomposition be released.