shishenghuang / iso-points

Code for Iso-Points: Optimizing Neural Implicit Surfaces with Hybrid Representations

Implementation for Iso-Points (CVPR 2021)

Official code for paper Iso-Points: Optimizing Neural Implicit Surfaces with Hybrid Representations

paper | supplementary material | project page

Overview

Iso-points are well-distributed points which lie on the neural iso-surface, they are an explicit form of representation of the implicit surfaces. We propose using iso-points to augment the optimization of implicit neural surfaces. The implicit and explicit surface representations are coupled, i.e. the implicit model determines the locations and normals of iso-points, whereas the iso-points can be utilized to control the optimization of the implicit model.

The implementation of the key steps for iso-points extraction is in levelset_sampling.py and utils/point_processing.py. To demonstrate the utilisation of iso-points, we provide scripts for multiple applications and scenarios:

• multiview reconstruction
  • Demo 1: importance sampling with iso-points
  • Demo 2: Multiview Neural Surface Reconstruction on DTU data with iso-points
• surface reconstruction from sparse point cloud

Demo

Installation

This code is built as an extension of out Differentiable Surface Splatting pytorch library (DSS), which depends on pytorch3d, torch_cluster. Currently we support up to pytorch 1.6.

git clone --recursive https://github.com/yifita/iso-points.git
cd iso-points

# conda environment and dependencies
# update conda
conda update -n base -c defaults conda
# install requirements
conda env create --name DSS -f environment.yml
conda activate DSS

# build additional dependencies of DSS
# FRNN - fixed radius nearest neighbors
cd external/FRNN/external
git submodule update --init --recursive
cd prefix_sum
python setup.py install
cd ../..
python setup.py install

# build batch-svd
cd ../torch-batch-svd
python setup.py install

# build DSS itself
cd ../..
python setup.py develop

prepare data

Download data

cd data
wget https://igl.ethz.ch/projects/iso-points/data.zip
unzip data.zip
rm data.zip

Including subset of masked DTU data (courtesy of Yariv et.al.), synthetic rendered multiview data, and masked furu stereo reconstruction of DTU dataset.

multiview reconstruction

sampling-with-iso-points

# train baseline implicit representation only using ray-tracing
python train_mvr.py configs/compressor_implicit.yml --exit-after 6000

# train with uniform iso-points
python train_mvr.py configs/compressor_uni.yml --exit-after 6000

# train with iso-points distributed according to loss value (hard example mining)
python train_mvr.py configs/compressor_uni_lossS.yml --exit-after 6000

DTU-data

python train_mvr.py configs/dtu55_iso.yml

implicit surface to noisy point cloud

python test_dtu_points.py data/DTU_furu/scan122.ply --use_off_normal_loss -o exp/points_3d_outputs/scan122_ours

cite

Please cite us if you find the code useful!

@inproceedings{yifan2020isopoints,
      title={Iso-Points: Optimizing Neural Implicit Surfaces with Hybrid Representations},
      author={Wang Yifan and Shihao Wu and Cengiz Oztireli and Olga Sorkine-Hornung},
      year={2020},
      booktitle = {CVPR},
      year = {2020},
}

Acknowledgement

This work was supported in parts by Apple scholarship, SWISSHEART Failure Network (SHFN), and UKRI Future Leaders Fellowship [grant number MR/T043229/1]

A good portion of this codebase uses or adapts codes from previous works and implementations. We sincerely thank the authors for their effort in making their work accessible. Most notably we refer to the following repos

• Siren: https://github.com/vsitzmann/siren
• IDR: https://github.com/lioryariv/idr
• DVR: https://github.com/autonomousvision/differentiable_volumetric_rendering