zju3dv / neusc

A Temporal Voyage: Code for "Neural Scene Chronology" [CVPR 2023]

Neural Scene Chronology

Project Page | Video | Paper | Data

Neural Scene Chronology
Haotong Lin, Qianqian Wang, Ruojin Cai, Sida Peng, Hadar Averbuch-Elor, Xiaowei Zhou, Noah Snavely
CVPR 2023

Installation

Set up the python environment

conda create -n nsc python=3.10
conda activate nsc
conda install conda install pytorch==1.12.1 torchvision==0.13.1 torchaudio==0.12.1 cudatoolkit=11.3 -c pytorch
pip install -r requirements.txt
pip install git+https://github.com/haotongl/kaolin.git@ht 

Set up datasets

0. Set up workspace

The workspace is the disk directory that stores datasets, training logs, checkpoints and results. Please ensure it has enough space.

export workspace=$PATH_TO_YOUR_WORKSPACE

1. Pre-trained model

Download the pretrained model from this link.

Or you can use the following command to download it.

cd $workspace
mkdir trained_model/nsc/5pointz/base
cd trained_model/nsc/5pointz/base
gdown 1edfa_pYk1m_wxC7dmiHcs40BA89z-kW6

2. Chronology dataset (Optional; download it if you want do training)

Download the 5PointZ dataset from this link.

Or you can use the following command to download it.

cd $workspace
mkdir chronology
gdown 1ytPDh5s5bzVnPLU01jcQ4xOA9_DDgliD 
unzip 5pointz.zip -d chronology
rm 5pointz.zip

3. Running neural scene chronology on your custom chronology dataset

Make sure you have prepared images with timestamps. The timestamp refers to the prefix of the image; please refer to the 5PointZ example dataset.

1. The first step is to run colmap SfM, and undistort the images, followed by dense reconstruction to obtain meshed-poisson.ply. Use Meshlab to select the region of interest from meshed-poisson.ply, and save it as meshed-poisson-clean.ply.

2. Generate semantic maps. The purpose of this step is to segment the sky for training the environment map, and to segment pedestrians and vehicles to avoid these pixels during training. Please download the semantic model from this link.

python scripts/semantic/prepare_data.py --root_dir ${CUSTOM_DATA_PATH} --gpu 0 

3. Generate annots.

First, generate cam_dict.npy and train.txt, which store the camera pose and training list, respectively. You can modify train.txt to select specific time periods, or keep some images for testing purposes.

python scripts/colmap/gen_annots.py --input ${CUSTOM_DATA_PATH}

Then, generate trash.txt to filter out some of the more noisy images. For example, images where more than two-thirds of the pixels are portraits, or images with very few points registered during the SfM process. These images usually provide little help to the model.

python scripts/colmap/gen_trash.py --data_root ${CUSTOM_DATA_PATH}

Inference and Training

Reproducing the demo video

The following command will render the demo video with the specified camera path and time.

python run.py --type evaluate --cfg_file configs/exps/nsc/5pointz_renderdemo.yaml test_dataset.input_ratio 1.

Training a model

python train_net.py --cfg_file configs/exps/nsc/5pointz.yaml

Our code also supports multi-gpu training. The published pretrained model was trained for 400k iterations with 2 A6000 GPUs.

python -m torch.distributed.launch --nproc_per_node=2 train_net.py --cfg_file configs/exps/nsc/5pointz.yaml distributed True gpus 0,1

Visualization

python run.py --type evaluate --cfg_file configs/exps/nsc/5pointz.yaml save_result True

Citation

If you find this code useful for your research, please use the following BibTeX entry.

@inproceedings{lin2023neural,
  title={Neural Scene Chronology},
  author={Lin, Haotong and Wang, Qianqian and Cai, Ruojin and Peng, Sida and Averbuch-Elor, Hadar and Zhou, Xiaowei and Snavely, Noah},
  booktitle={CVPR},
  year={2023}
}

Acknowledgement

We would like to thank Shangzhan Zhang for the helpful disscussion. Some of the code releated to NGP in this repo is borrowed from torch-ngp, thanks Jiaxiang Tang!

neusc's Issues

How to run colmap sfm on a collection of images with different timestamps

In my understanding, sparse reconstruction models vary for images captured at different timestamps. I'm not understand about the process of running COLMAP to execute Structure-from-Motion (SfM) on an image dataset. Should all images with different timestamps be placed in a single folder？

Could you upload requirements.txt ?

The repository has no requirements.txt file

Could you please upload it?

Thank you

Could you please help update the README file? I plan to use my own data for training the scene renderer. Thank you very much for your assistance!

Could you please help update the README file? I plan to use my own data for training the scene renderer. Thank you very much for your help!

Code Release

Hi Haotong,
Thank you for the great work.
Any estimate on the code release date ?

How does Colmap perform dense reconstruction with a collection of images captured at different timestamps?

When using Colmap for reconstruction, images captured at the same location but at different timestamps can vary. How can dense reconstruction be achieved in this case, and will the point cloud models from different time points overlap?
I ran colmap for dense reconstruction and the patterns that were supposed to be in the same position were not reconstructed to the same position. Instead, it was rebuilt to another location.

'NeuSC/configs/opts/' directory when running my custom chronology dataset

I'd like to inquire about how to handle the three files located in the 'NeuSC/configs/opts/' directory when running my custom chronology dataset. Can you please provide guidance on how to modify or address these files appropriately

data processing scripts

I'm interested in obtaining the data processing scripts mentioned in the paper. Could you please provide information on where I can find or access these scripts?
Thank you very much

May I ask which version of MMSegmentation in scripts/semantic/prepare_data.py is used? I encountered an error during operation.

ImportError: cannot import name 'inference_segmentor' from 'mmseg.apis' . How to solve it.