GithubHelp home page GithubHelp logo

jingsenzhu / i2-sdf Goto Github PK

View Code? Open in Web Editor NEW
173.0 11.0 10.0 830 KB

[CVPR 2023] I^2-SDF: Intrinsic Indoor Scene Reconstruction and Editing via Raytracing in Neural SDFs

License: MIT License

Python 100.00%

i2-sdf's Introduction

News

  • 04/04/2023 dataset preview release: 2 synthetic scenes available
  • 15/04/2023 code release: 3D reconstruction and novel view synthesis part
  • 21/04/2023 dataset release: real data

TODO

  • Full dataset release
  • Code release for 3D reconstruction and novel view synthesis
  • Code release for intrinsic decomposition and scene editing

Dataset released

  • Synthetic: kitchen_0, bedroom_relight_0, bedroom_0, bedroom_1, bedroom_relight_1, diningroom_0, livingroom_0, livingroom_1, more scenes to be released
  • Real: inria_livingroom, nisr_livingroom, nisr_coffee_shop_0, nisr_coffee_shop_1, release complete

I2-SDF: Intrinsic Indoor Scene Reconstruction and Editing via Raytracing in Neural SDFs (CVPR 2023)

Setup

Installation

conda env create -f environment.yml
conda activate i2sdf

Data preparation

Download our synthetic dataset and extract them into data/synthetic. If you want to run on your customized dataset, we provide a brief introduction to our data convention here.

Dataset

We provide a high-quality synthetic indoor scene multi-view dataset, with ground truth camera pose and geometry annotations. See HERE for data conventions. Click HERE to download.

3D Reconstruction and Novel View Synthesis

Training

python main_recon.py --conf config/<config_file>.yml --scan_id <scan_id> -d <gpu_id> -v <version>

Note: config/synthetic.yml doesn't contain light mask network, while config/synthetic_light_mask.yml contains.

If you run out of GPU memory, try to reduce the split_n_pixels (i.e. validation batch size), batch_size in the config. The default parameters are evaluated under RTX A6000 (48GB). For RTX 3090 (24GB), try to set split_n_pixels 5000.

Evaluation

Novel view synthesis

python main_recon.py --conf config/<config_file>.yml --scan_id <scan_id> -d <gpu_id> -v <version> --test [--is_val] [--full]

The optional flag --is_val evaluates on the validation set instead of training set, --full produces full-resolution rendered images without downsampling.

View Interpolation

python main_recon.py --conf config/<config_file>.yml --scan_id <scan_id> -d <gpu_id> -v <version> --test --test_mode interpolate --inter_id <view_id_0> <view_id_1> [--full]

Generates a view interpolation video between 2 views. Requires ffmpeg being installed.

The number of frames and frame rate of the video can be specified by options.

Mesh Extraction

python main_recon.py --conf config/<config_file>.yml --scan_id <scan_id> -d <gpu_id> -v <version> --test --test_mode mesh

Intrinsic Decomposition and Scene Editing

Brewing🍺, code coming soon.

Citation

If you find our work is useful, please consider cite:

@inproceedings{zhu2023i2sdf,
    title = {I$^2$-SDF: Intrinsic Indoor Scene Reconstruction and Editing via Raytracing in Neural SDFs},
    author = {Jingsen Zhu and Yuchi Huo and Qi Ye and Fujun Luan and Jifan Li and Dianbing Xi and Lisha Wang and Rui Tang and Wei Hua and Hujun Bao and Rui Wang},
    booktitle = {CVPR},
    year = {2023}
}

Acknowledgement

i2-sdf's People

Contributors

jingsenzhu avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

i2-sdf's Issues

Dataset camera intrinsics

Hi,

Thanks for sharing this work. Does the dataset contain any camera intinsics information, such as focal length / fov, optical center, etc? It seems like cameras.npz contains only the camera to world transformation matrix.

Thanks!

Material labels for released dataset?

Hi there, thanks for releasing the two demo scenes! I wonder if there is a plan to include materials and emitter mask in the current/later version (as indicated here), which will be useful for inverse rendering tasks.

Thanks!

what's depth/normal supervision ?

Hi,

Great work. I have questions regarding equation 9) in the paper.
What are the depth/normal supervision ? specifically, what are D(r) and N(r) in 11) and 12) respectively.

  1. monosdf uses public models to generate depth maps and normal maps to supervise the model. Just curious how to generate depth/normal supervision for i2sdf.
  2. Do i2sdf and monosdf share the same depth/normal supervision ?

Thanks

Artifacts with depth maps?

Hi there I noticed some potential issues with a few depth maps, where pixels that belong to a open window have large depth values, instead of 0. Here is the 0003 image of the released bedroom scene.
aad0bf13-c07f-4e3e-aeca-a79bd31af370

Note the yellow strip in the center figure, which are inside the white area of the 'window hole'; the pixels have large depth values but ideally should be infinite (0) because they correspond to the outdoor env. I pick two locations (red dot and green dot) on the figures, and query the depths, getting 4.3840003 3.1920002 respectively, but the former one should be 0.

As a result, if you simply back-project depth to 3D, you get fantom geometry (in red circle).

CleanShot 2023-04-13 at 23 09 24@2x

Not sure if this is an issue with the way I read the depth file (cv2.imread(filename, -1)), or the depth file itself.

Some weired floater in the extracted mesh

Thanks for sharing the great work!
But when I ran the mesh extract for scan 31 and scan 127 I got very weired look, like the following:
image
image

The main part of the scan looks great, but there still some floater or redundant plane. Do you have any idea about this?
The environment information is:
tinycudann 1.7
torch 1.13.1+cu117
Ubuntu 22.04.2 LTS
NVIDIA 4090
Driver Version: 525.116.04

When the code will be released?

Hello. Thank you for sharing this amazing project!

May I know when the code will be released?

I am looking forward to running this project!

Inconsistent normal losses

I wanted to let you know about an inconsistency in the code.

The code seems to contain two normal losses:

  1. get_normal_l1_loss, which actually computes an angular loss, as pointed out in Equation 12 of your paper. This name seems confusing, as it does not compute the L1 normal loss.
  2. get_normal_angular_loss, which actually computes the truncated scaled angle.

However, the loss computation in forward uses loss 1 above twice (see this line), weighted by (self.normal_weight + self.angular_weight), and loss 2 above is not used at all.

Missing config file

Hi,
Great work and thanks for sharing! I wanted to test your code after your last update, however, I can't find an example of a config file or documentation for it. Is this something you could provide?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.