xharlie / pointnerf

Point-NeRF: Point-based Neural Radiance Fields

Point-NeRF: Point-based Neural Radiance Fields (CVPR 2022 Oral 🤩)

Project Sites | Paper | Primary contact: Qiangeng Xu

Point-NeRF uses neural 3D point clouds, with associated neural features, to model a radiance field. Point-NeRF can be rendered efficiently by aggregating neural point features near scene surfaces, in a ray marching-based rendering pipeline. Moreover, Point-NeRF can be initialized via direct inference of a pre-trained deep network to produce a neural point cloud; this point cloud can be finetuned to surpass the visual quality of NeRF with 30X faster training time. Point-NeRF can be combined with other 3D reconstruction methods and handles the errors and outliers in such methods via a novel pruning and growing mechanism.

Reference

Please cite our paper if you are interested
Point-NeRF: Point-based Neural Radiance Fields.    

@inproceedings{xu2022point,
  title={Point-nerf: Point-based neural radiance fields},
  author={Xu, Qiangeng and Xu, Zexiang and Philip, Julien and Bi, Sai and Shu, Zhixin and Sunkavalli, Kalyan and Neumann, Ulrich},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  pages={5438--5448},
  year={2022}
}

Updates

1. To replace pycuda, we have implemented the pytorch cuda functions when using world coordinates to group neural points. Simply set wcoord_query=-1 in your configuration file if the original setting is wcoord_query=1 (see dev_scripts/w_n360/chair_cuda.sh).
2. We have received constructive feedbacks that when Point-NeRF use MVSNet to reconstruct point cloud, the point fusion after depth estimation by MVSNet will use the alpha channel information in the NeRF-Synthetic Dataset. It is due to the fact that MVSNet cannot handle background very well. To improve the fairness, we include new training scripts and results of PointNeRF + MVSNet when using background color for filtering. The results (see below) are similar to the ones that are previously reported.

	Chair	Drums	Lego	Mic	Materials	Ship	Hotdog	Ficus	Avg
PSNR	35.60	26.04	35.27	35.91	29.65	30.61	37.34	35.61	33.25
SSIM	0.991	0.954	0.989	0.994	0.971	0.938	0.991	0.992	0.978
LPIPSVgg	0.023	0.078	0.021	0.014	0.071	0.129	0.036	0.025	0.050
LPIPSAlex	0.010	0.055	0.010	0.007	0.041	0.076	0.016	0.011	0.028

This issue only affacts situations when Point-NeRF uses MVSNet on NeRF-Synthetic Dataset. The Colmap results and results on other datasets are not impacted.
An even more reasonable reconstruction approach should exclude using the knowledge of background color or other point filtering. Therefore, we suggest users to combine PointNeRF with more powerful MVS models, such as TransMVS.

Overal Instruction

1. Please first install the libraries as below and download/prepare the datasets as instructed.
2. Point Initialization: Download pre-trained MVSNet as below and train the feature extraction from scratch or directly download the pre-trained models. (Obtain 'MVSNet' and 'init' folder in checkpoints folder)
3. Per-scene Optimization: Download pre-trained models or optimize from scratch as instructed.

For nerfsynthetic, colmap_nerfsynthetic, tanks&temples, scannet and dtu,
We provide all the checkpoint_files google drive | baidu wangpan; all the images and scores of the test results google drive | baidu wangpan; and video results google drive | baidu wangpan;

We also share the visual results of npbg, nsvf and ibrnet on the Nerf Synthetic dataset generated by our machine google drive; Please cite their papers accordingly if interested.

Installation

Requirements

All the codes are tested in the following environment:

• Linux (tested on Ubuntu 16.04, 18.04, 20.04)
• Python 3.6+
• PyTorch 1.7 or higher (tested on PyTorch 1.7, 1.8.1, 1.9, 1.10)
• CUDA 10.2 or higher

Install

Install the dependent libraries as follows:

• Install the dependent python libraries:

pip install torch==1.8.1+cu102 h5py
pip install imageio scikit-image

• Install pycuda (crucial) following: https://documen.tician.de/pycuda/install.html
• Install torch_scatter following: https://github.com/rusty1s/pytorch_scatter

We develope our code with pytorch1.8.1, pycuda2021.1, and torch_scatter 2.0.8

Data Preparation

The layout should looks like this, we provide all data folder here: google_drive, except for scannet (We take it down on Mar 8th 2023, to respect Scannet's policy, please go to Scannet's official website for data.)

pointnerf
├── data_src
│   ├── dtu
    │   │   │──Cameras
    │   │   │──Depths
    │   │   │──Depths_raw
    │   │   │──Rectified
    ├── nerf
    │   │   │──nerf_synthetic
    │   │   │──nerf_synthetic_colmap
    ├── TanksAndTemple
    ├── scannet
    │   │   │──scans 
    |   │   │   │──scene0101_04
    |   │   │   │──scene0241_01

Or you can download using the official links as follows:

DTU:

Download the preprocessed DTU training data and Depth_raw from original MVSNet repo and unzip.

NeRF Synthetic

Download nerf_synthetic.zip from here under ``data_src/nerf/''

Tanks & Temples

Follow Neural Sparse Voxel Fields and download Tanks&Temples | download (.zip) | 0_* (training) 1_* (testing) as: ``data_src/TanksAndTemple/''

ScanNet

Download and extract ScanNet by following the instructions provided at http://www.scan-net.org/. The detailed steps including:

• Go to http://www.scan-net.org and fill & sent the request form.
• You will get a email that has command instruction and a download-scannet.py file, this file is for python 2, you can use our download-scannet.py in the ``data'' directory for python 3.
• clone the official repo:

  git clone https://github.com/ScanNet/ScanNet.git
  

• Download specific scenes (used by NSVF):

   python data/download-scannet.py -o ../data_src/scannet/ id scene0101_04 
   python data/download-scannet.py -o ../data_src/scannet/ id scene0241_01
  

• Process the sens files:

    python ScanNet/SensReader/python/reader.py --filename data_src/nrData/scannet/scans/scene0101_04/scene0101_04.sens  --output_path data_src/nrData/scannet/scans/scene0101_04/exported/ --export_depth_images --export_color_images --export_poses --export_intrinsics
    
    python ScanNet/SensReader/python/reader.py --filename data_src/nrData/scannet/scans/scene0241_01/scene0241_01.sens  --output_path data_src/nrData/scannet/scans/scene0241_01/exported/ --export_depth_images --export_color_images --export_poses --export_intrinsics
  

Point Initialization / Generalization:

  Download pre-trained MVSNet checkpoints:

We trained MVSNet on DTU. You can Download ''MVSNet'' directory from google drive and place them under '''checkpoints/'''

  Train 2D feature extraction and point representation

  Directly use our trained checkpoints files:

Download ''init'' directory from google drive. and place them under '''checkpoints/'''

  Or train from scratch:

Train for point features of 63 channels (as in paper)

bash dev_scripts/ete/dtu_dgt_d012_img0123_conf_color_dir_agg2.sh

Train for point features of 32 channels (better for per-scene optimization)

bash dev_scripts/ete/dtu_dgt_d012_img0123_conf_agg2_32_dirclr20.sh

After the training, you should pick a checkpoint and rename it to best checkpoint, e.g.:

cp checkpoints/dtu_dgt_d012_img0123_conf_color_dir_agg2/250000_net_ray_marching.pth  checkpoints/dtu_dgt_d012_img0123_conf_color_dir_agg2/best_net_ray_marching.pth

cp checkpoints/dtu_dgt_d012_img0123_conf_color_dir_agg2/250000_net_mvs.pth  checkpoints/dtu_dgt_d012_img0123_conf_color_dir_agg2/best_net_mvs.pth

  Test feed forward inference on dtu scenes

These scenes that are selected by MVSNeRF, please also refer their code to understand the metrics calculation.

bash dev_scripts/dtu_test_inf/inftest_scan1.sh
bash dev_scripts/dtu_test_inf/inftest_scan8.sh
bash dev_scripts/dtu_test_inf/inftest_scan21.sh
bash dev_scripts/dtu_test_inf/inftest_scan103.sh
bash dev_scripts/dtu_test_inf/inftest_scan114.sh

Per-scene Optimization:

(Please visit the project sites to see the original videos of above scenes, which have quality loss when being converted to gif files here.)

Download per-scene optimized Point-NeRFs

You can skip training and download the folders of ''nerfsynth'', ''tanksntemples'' and ''scannet'' here google drive, and place them in ''checkpoints/''.

pointnerf
├── checkpoints
│   ├── init
    ├── MVSNet
    ├── nerfsynth
    ├── col_nerfsynth
    ├── scannet
    ├── tanksntemples

In each scene, we provide initialized point features and network weights ''0_net_ray_marching.pth'', points and weights at 20K steps ''20000_net_ray_marching.pth'' and 200K steps ''200000_net_ray_marching.pth''

Test the per-scene optimized Point-NeRFs

NeRF Synthetics

test scripts

    bash dev_scripts/w_n360/chair_test.sh
    bash dev_scripts/w_n360/drums_test.sh
    bash dev_scripts/w_n360/ficus_test.sh
    bash dev_scripts/w_n360/hotdog_test.sh
    bash dev_scripts/w_n360/lego_test.sh
    bash dev_scripts/w_n360/materials_test.sh
    bash dev_scripts/w_n360/mic_test.sh
    bash dev_scripts/w_n360/ship_test.sh

ScanNet

test scripts

    bash dev_scripts/w_scannet_etf/scane101_test.sh
    bash dev_scripts/w_scannet_etf/scane241_test.sh

Tanks & Temples

test scripts

    bash dev_scripts/w_tt_ft/barn_test.sh
    bash dev_scripts/w_tt_ft/caterpillar_test.sh
    bash dev_scripts/w_tt_ft/family_test.sh
    bash dev_scripts/w_tt_ft/ignatius_test.sh
    bash dev_scripts/w_tt_ft/truck_test.sh

Per-scene optimize from scatch

Make sure the ''checkpoints'' folder has ''init'' and ''MVSNet''. The training scripts will start to do initialization if there is no ''.pth'' files in a scene folder. It will start from the last ''.pth'' files until reach the iteration of ''maximum_step''.

NeRF Synthetics using MVSNet (w/ alpha channel filtering during point cloud reconstruction and pycuda)

train scripts

    bash dev_scripts/w_n360/chair.sh
    bash dev_scripts/w_n360/drums.sh
    bash dev_scripts/w_n360/ficus.sh
    bash dev_scripts/w_n360/hotdog.sh
    bash dev_scripts/w_n360/lego.sh
    bash dev_scripts/w_n360/materials.sh
    bash dev_scripts/w_n360/mic.sh
    bash dev_scripts/w_n360/ship.sh

NeRF Synthetics using MVSNet (w/ background color filtering during point cloud reconstruction and pytorch cuda)

train scripts

    bash dev_scripts/w_n360/chair_cuda.sh
    bash dev_scripts/w_n360/drums_cuda.sh
    bash dev_scripts/w_n360/ficus_cuda.sh
    bash dev_scripts/w_n360/hotdog_cuda.sh
    bash dev_scripts/w_n360/lego_cuda.sh
    bash dev_scripts/w_n360/materials_cuda.sh
    bash dev_scripts/w_n360/mic_cuda.sh
    bash dev_scripts/w_n360/ship_cuda.sh

NeRF Synthetics using COLMAP points

Please download the COLMAP data (see above). If there is {maximum_step}.pth checkpoint files in the path, the scripts below will also run test.

train scripts

    bash dev_scripts/w_colmap_n360/col_chair.sh
    bash dev_scripts/w_colmap_n360/col_drums.sh
    bash dev_scripts/w_colmap_n360/col_ficus.sh
    bash dev_scripts/w_colmap_n360/col_hotdog.sh
    bash dev_scripts/w_colmap_n360/col_lego.sh
    bash dev_scripts/w_colmap_n360/col_materials.sh
    bash dev_scripts/w_colmap_n360/col_mic.sh
    bash dev_scripts/w_colmap_n360/col_ship.sh

ScanNet

train scripts

    bash dev_scripts/w_scannet_etf/scene101.sh
    bash dev_scripts/w_scannet_etf/scene241.sh

Tanks & Temples

train scripts

    bash dev_scripts/w_tt_ft/barn.sh
    bash dev_scripts/w_tt_ft/caterpillar.sh
    bash dev_scripts/w_tt_ft/family.sh
    bash dev_scripts/w_tt_ft/ignatius.sh
    bash dev_scripts/w_tt_ft/truck.sh

Acknowledgement

Our repo is developed based on MVSNet, NeRF, MVSNeRF, and NSVF.

Please also consider citing the corresponding papers.

The project is conducted collaboratively between Adobe Research and University of Southern California.

LICENSE

The repo is licensed under Creative Commons Attribution-NonCommercial-ShareAlike 2.0, and is restricted to academic use only. See LICENSE.

pointnerf's Issues

Generate depth map

Thank you for your work and contribution!
I want to generate a depth map, in the "neural_points_volumetric_model.py " file, I set the self.return_depth is True, however, there is a warning that 'ray_ts' is not defined. And I want to know what is the 'ray_ts' in your code, looking forward to your reply.

pycuda.driver.CompileError

Hello,
Thanks for the awesome work. However when I run the Per-scene Optimization on NeRF-Synthetics I found some error below:

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Debug Mode
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
/home/zhaoboming/anaconda3/envs/nerf-w/lib/python3.7/site-packages/numpy/core/shape_base.py:420: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray.
arrays = [asanyarray(arr) for arr in arrays]
/home/zhaoboming/anaconda3/envs/nerf-w/lib/python3.7/site-packages/torch/functional.py:568: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at /opt/conda/conda-bld/pytorch_1646755953518/work/aten/src/ATen/native/TensorShape.cpp:2228.)
return _VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined]
dataset total: train 100
dataset [NerfSynthFtDataset] was created
../checkpoints/nerfsynth/lego/*_net_ray_marching.pth
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Continue training from 200000 epoch
Iter: 200000
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
opt.act_type!!!!!!!!! LeakyReLU
self.points_embeding torch.Size([1, 479862, 32])
querier device cuda:3 3
Traceback (most recent call last):
File "train_ft.py", line 1084, in
main()
File "train_ft.py", line 637, in main
model = create_model(opt)
File "/mnt/data1/zhaoboming/pointnerf/run/../models/init.py", line 39, in create_model
instance.initialize(opt)
File "/mnt/data1/zhaoboming/pointnerf/run/../models/base_rendering_model.py", line 369, in initialize
self.create_network_models(opt)
File "/mnt/data1/zhaoboming/pointnerf/run/../models/mvs_points_volumetric_model.py", line 44, in create_network_models
super(MvsPointsVolumetricModel, self).create_network_models(opt)
File "/mnt/data1/zhaoboming/pointnerf/run/../models/neural_points_volumetric_model.py", line 157, in create_network_models
params = self.get_additional_network_params(opt)
File "/mnt/data1/zhaoboming/pointnerf/run/../models/neural_points_volumetric_model.py", line 142, in get_additional_network_params
self.neural_points = NeuralPoints(opt.point_features_dim, opt.num_point, opt, self.device, checkpoint=checkpoint_path, feature_init_method=opt.feature_init_method, reg_weight=0., feedforward=opt.feedforward)
File "/mnt/data1/zhaoboming/pointnerf/run/../models/neural_points/neural_points.py", line 331, in init
self.querier = self.lighting_fast_querier(device, self.opt)
File "/mnt/data1/zhaoboming/pointnerf/run/../models/neural_points/query_point_indices_worldcoords.py", line 39, in init
self.claim_occ, self.map_coor2occ, self.fill_occ2pnts, self.mask_raypos, self.get_shadingloc, self.query_along_ray = self.build_cuda()
File "/mnt/data1/zhaoboming/pointnerf/run/../models/neural_points/query_point_indices_worldcoords.py", line 524, in build_cuda
""", no_extern_c=True)
File "/home/zhaoboming/anaconda3/envs/nerf-w/lib/python3.7/site-packages/pycuda-2021.1-py3.7-linux-x86_64.egg/pycuda/compiler.py", line 358, in init
include_dirs,
File "/home/zhaoboming/anaconda3/envs/nerf-w/lib/python3.7/site-packages/pycuda-2021.1-py3.7-linux-x86_64.egg/pycuda/compiler.py", line 298, in compile
return compile_plain(source, options, keep, nvcc, cache_dir, target)
File "/home/zhaoboming/anaconda3/envs/nerf-w/lib/python3.7/site-packages/pycuda-2021.1-py3.7-linux-x86_64.egg/pycuda/compiler.py", line 87, in compile_plain
checksum.update(preprocess_source(source, options, nvcc).encode("utf-8"))
File "/home/zhaoboming/anaconda3/envs/nerf-w/lib/python3.7/site-packages/pycuda-2021.1-py3.7-linux-x86_64.egg/pycuda/compiler.py", line 59, in preprocess_source
"nvcc preprocessing of %s failed" % source_path, cmdline, stderr=stderr
pycuda.driver.CompileError: nvcc preprocessing of /tmp/tmpeytiys5x.cu failed
[command: nvcc --preprocess -arch sm_86 -I/mnt/data1/zhaoboming/anaconda3/envs/nerf-w/lib/python3.7/site-packages/pycuda-2021.1-py3.7-linux-x86_64.egg/pycuda/cuda /tmp/tmpeytiys5x.cu --compiler-options -P]
[stderr:
b'cc1plus: fatal error: cuda_runtime.h: No such file or directory\ncompilation terminated.\n']
end loading

---

PyCUDA ERROR: The context stack was not empty upon module cleanup.

A context was still active when the context stack was being
cleaned up. At this point in our execution, CUDA may already
have been deinitialized, so there is no way we can finish
cleanly. The program will be aborted now.
Use Context.pop() to avoid this problem.

So How can I fix it? Thanks.

Training MVS Net for Depth Prediction

Hi Xharlie,

thank you very much for publishing your code!

I try to apply your method to some own data and I would like to train the MVSNet on it. Could you please provide some guidance on how to do this with your repo?

Your help is very much appreciated.

Questions about Initializing Neural Points from COLMAP Points

Hello, I just meet some problem when checking the code of intializing neural points from COLMAP points.
In your paper, you said that

We first rule out all the views where the point is occluded by other points, then we find the view of which the camera is the closest to the point.

,but I can't find the code of rule out all the views where the point is occluded by other points. In line 709 - line 720 of train_ft.py, you just find the the view of which the camera is the closest to the point.

I'm also troubled by the formula for calculating the distance between camera and view which is implemented as

dists_norm = torch.norm(dists, dim=-1) # N, M
dists_dir = dists / (dists_norm[...,None]+1e-6) # N, M, 3
dists = dists_norm / 200 + (1.1 - torch.sum(dists_dir * raydir[None, :],dim=-1))

I wonder that what does number 200 and 1.1 mean in this formula. And how does this formula ensures that the point is on the camera plane(the pixel coordinates of the point is between [0,img_size])?

dataset

Hi.
Could you also upload the dtu dataset to baidu pan, I can't download it from google drive anyway, it's too big.

Extract mesh ?

Have you provided extracting mesh function?

Error on inference without per-scene optimization

Hello authors, thank you very much for your paper and for releasing code! I am very excited about this work.

I am trying to run inference on a model that has been trained for point initialization (not with per-scene optimization). The README indicates that it should be possible to perform inference on DUT with bash dev_scripts/dtu_test_inf/inftest_scan8.sh.

When I run this script, I get an error:

File "/path/to/pointnerf/run/../models/neural_points_volumetric_model.py", line 270, in forward
    decoded_features, ray_valid, weight, conf_coefficient = self.aggregator(sampled_color, sampled_Rw2c, sampled_dir, sampled_conf, sampled_embedding, sampled_xyz_pers, sampled_xyz, sample_pnt_mask, sample_loc, sample_loc_w, sample_ray_dirs, vsize, grid_vox_sz)
...
File "/path/to/python3.8/site-packages/torch/nn/functional.py", line 1847, in linear
    return torch._C._nn.linear(input, weight, bias)
RuntimeError: CUDA error: CUBLAS_STATUS_EXECUTION_FAILED when calling `cublasSgemm( handle, opa, opb, m, n, k, &alpha, a, lda, b, ldb, &beta,
 c, ldc)`

I am using CUDA 11.1 and I believe that everything CUDA-related is installed correctly.

I was also confused by something else: this script calls train_ft.py, not test.py as I would have expected. In fact, there is not a single script in dev_scripts that calls test.py. Am I misunderstanding something here, or is test.py actually never used?

"no such file or directory " pointcloud

Hello . Thanks for your amazing work.

However , When I run bash/dev_scripts/hotdog_test.sh ， it goes with FileNotFoundError ,in line 248 load_blender_cloud load_blender_cloud.py , where is the pointcloud saved/dumped in your scripts . I can't find the this.

Clarification about Tanks & Temples results

Hi @Xharlie and thanks for the awesome work. I have been trying to bias NeRF with geometry lately and I'm glad to see such a good work published at CVPR 2022.

I have a question about the evaluation process on Tanks & Temples. Let's take the truck scene as an example. You report a PSNR of 25.36 for basic NeRF, that actually matches the value in NSVF (I assume it's just taken from there). In that paper, they say: "We label the object masks ourselves with the software of Altizure".

Do you also use object masks and compute the metrics only on pixels from that mask? It seems plausible, as the results shown in your paper and your repo contain only the object, without the background.

I'm asking this because the PSNR of NeRF is significantly smaller, when computed on the whole image. It drops down to 20.85 as reported in NeRF++ and I have been able to reach a bit more than 21 with some tuning, but not more.

Thank you in advance for the answer!

Any instruction for our own data preparation?

Hi there, I followed all of your instructions and now can train from the scratch. However, I met some difficulties when trying point nerf on my own data. I think some more instructions may be needed for data preparation. Thanks a lot!

On which scenes the point growing fails?

Hi, @Xharlie

Thanks for your interesting work and for sharing the code. I love this work pretty much. While I noticed there is a point growing module of the network and Figure. 5 in the paper shows the module can grow points from extremely sparse point clouds, then why do we need another MVS Network to predict dense point clouds? Since we can obtain sparse point clouds from differentiable triangulation if camera poses are given. I would also like to know if there are any ablation studies showing the difference. Besides, it's interesting to show on which cases the point growing module will fail.

"No such file or directory" only for scannet scene

Hi , when I redownload the code repo and run from scratch ,It still encounters with the problem of "No such file or directory" when I run scene101.sh ,However it works fine if I run from scratch in the dataset of Nerf Synth. Is there some possiblity that I should change some setting for the scannet scene.
#7 (comment)

release of full environment setup

Dear authro,

Can you release the full environment setup?
such as environment file in conda.

Thanks

About

Thanks for sharing the code! PointNerf's synthesis result is really amazing!

I've read the code and confused about how to compute the ray_dist. In vanilla NeRF, the sigma is usually computed using ray-distance, but PointNerf seems using depth instead of ray distance to compute sigma?

pointnerf/models/neural_points_volumetric_model.py

Line 271 in 9377094

ray_dist = torch.cummax(sample_loc[..., 2], dim=-1)[0]

DTU dataset download

Hello,

Thanks for the awesome work. However, I am not able to download the DTU training dataset from the gdrive link shared in the repo. It seems that the link is dead. Would it be possible for you to reactivate the link?

Thanks,
Aditya Vora

FileNotFoundError of pointcloud

Hi, thanks for your great job! I have the same problem when I run "bash dev_scripts/w_n360/ship_test.sh". It goes with "FileNotFoundError: [Errno 2] No such file or directory" in line 118 of load_blender.py. My datapath is pointnerf/data_src/nerf/nerf_synthetic/ship，and it includes 3 .json files and 3 folders containing some .png pictures. At the same time, the checkpoints folder only include some .pth files. It doesn't seem to contain the saved point cloud.

Could you please tell me where the "point_path" is? Thank you~

Nerf LLFF dataset ?

Hi ,thanks for your amazing work.

However , I'm curious about the result in Nerf LLFF forward scenes as I haven't seen the result in your paper . Have you tested the performance on this data , is it still good on outdoor scene ? Sorry to trouble you for this ,because it is a bit difficult for me to reproduce the result on LLFF dataset.

Question about SR

Hello , sorry to bother again. I saw "--SR" is the max shading points number each ray, and when process the rays, the rays have more than 80 shading points are deleted.
I wonder why we need this upper bound setting, any reason behind it?

How to reconstruct nerf_synthetic data using colmap?

Hi, @Xharlie

Thanks for your code and data. I tried to rebuild the nerf_synthetic data using the default settings of colmap dense rebuild pipeline, but I found that the results are poor and the quality of the generated point cloud is far from the one you provided. Could you please share your reconstruction settings or provide an automatic reconstruction script?

For example, on the lego data, it is clear that the key points of the two sides of the tracks were matched together, causing an error in the reconstruction.

Loss

In base_rendering_model, loss = self.l2loss(masked_output, masked_gt) * masked_gt.shape[1], why multiply masked_gt.shape[1]?

pointnerf/models/base_rendering_model.py

Line 560 in a614e1d

loss = self.l2loss(masked_output, masked_gt) * masked_gt.shape[1]

About points color

Hi,
Thanks for the awesome work. However I have a doubt about the parameters Neural_points.points_color, why they are negative? So what does this 'color' represent?

How to use point cloud as input

Hello authors, thank you very much for your paper and for releasing code! I am very excited about this work.
I have gotten point cloud data in a ply format somehow, how do I use this data as input to point_nerf. I don't seem to find it in "READEME.md".

Google drive with evaluation results is missing DTU

Hi @Xharlie,

thanks for the great work! The link to scores and images on all the datasets is currently missing DTU dataset results. Can you please upload it? Thanks!

-Sergey

PyCUDA ERROR: The context stack was not empty upon module cleanup

After running ‘bash dev_scripts/w_scannet_etf/scane241_test.sh’ and 'bash dev_scripts/w_n360/chair_test.sh' I got this PyCUDA error:

Does anybody know how to solve it?

models

hi @Xharlie @zexxu

What's the difference of neural_points_volumetric_model.py and mvs_points_volumetric_model.py

best,
Yingjie

About the way sampling rays

Thanks for you brilliant work. I have a question about sampling rays. Since I want to render a patch, I set the parameter 'random_sample' in scene101.sh and in train_ft.py line 917 as 'patch', then run per-scene optimization on scannet scene0101_04. In this way, the performance is inferior (~2 drop on PSNR) to randomly sampling rays. Is my setup correct? Have you ever tried the patch sampling? Thanks

batch size can not bigger than 1 !?

Dear Author,

It seems the code does not supper batch_size >1? Because the code here?
https://github.com/Xharlie/pointnerf/blob/master/models/neural_points/query_point_indices_worldcoords.py#:~:text=R%20%3D%20torch.max(torch.sum(ray_mask_tensor.to(torch.int32))).cpu().numpy()

Bests

Error on gen_points_filter_embeddings()

Hello authors, thank you very much for your paper and for releasing code!

I am trying to run : bash ./dev_scripts/w_n360/ship.sh

When I run this script, I get the follow log and error:

-----------------------------------Generate Points-----------------------------------
model [MvsPointsVolumetricModel] was created
opt.resume_iter!!!!!!!!! best
loading mvs from ../checkpoints/init/dtu_dgt_d012_img0123_conf_agg2_32_dirclr20/best_net_mvs.pth
cannot load ../checkpoints/init/dtu_dgt_d012_img0123_conf_agg2_32_dirclr20/best_net_mvs.pth
------------------- Networks -------------------
[Network mvs] Total number of parameters: 0.382M

0%| | 0/543 [00:00<?, ?it/s]/usr/local/lib/python3.7/dist-packages/torch/nn/functional.py:3829: UserWarning: Default grid_sample and affine_grid behavior has changed to align_corners=False since 1.3.0. Please specify align_corners=True if the old behavior is desired. See the documentation of grid_sample for details.
"Default grid_sample and affine_grid behavior has changed "
100%|
543/543 [02:26<00:00, 3.71it/s]
100%|
543/543 [31:52<00:00, 3.52s/it]
xyz_world_all torch.Size([0, 3]) torch.Size([0, 1]) torch.Size([0])
%%%%%%%%%%%%% getattr(dataset, spacemin, None) None
vishull_mask torch.Size([0])
alpha masking xyz_world_all torch.Size([0, 3]) torch.Size([0, 1])
Traceback (most recent call last):
File "train_ft.py", line 1084, in
main()
File "train_ft.py", line 639, in main
points_xyz_all, points_embedding_all, points_color_all, points_dir_all, points_conf_all, img_lst, c2ws_lst, w2cs_lst, intrinsics_all, HDWD_lst = gen_points_filter_embeddings(train_dataset, visualizer, opt)
File "train_ft.py", line 142, in gen_points_filter_embeddings
xyz_world_all, sparse_grid_idx, sampled_pnt_idx = mvs_utils.construct_vox_points_closest(xyz_world_all.cuda() if len(xyz_world_all) < 99999999 else xyz_world_all[::(len(xyz_world_all)//99999999+1),...].cuda(), opt.vox_res)
File "/root/pointnerf/run/../models/mvs/mvs_utils.py", line 541, in construct_vox_points_closest
xyz_min, xyz_max = torch.min(xyz, dim=-2)[0], torch.max(xyz, dim=-2)[0]
RuntimeError: cannot perform reduction function min on tensor with no elements because the operation does not have an identity
end loading

I add some log to debug this issue as follow , found out: all item of confidence_all equals to 0.0208.

torch.set_printoptions(profile="full")
print(cam_xyz_all)
_, xyz_world_all, confidence_filtered_all = filter_utils.filter_by_masks_gpu(cam_xyz_all, intrinsics_all, extrinsics_all, confidence_all, points_mask_all, opt, vis=True, return_w=True, cpu2gpu=cpu2gpu, near_fars_all=near_fars_all)

Why is it slow for Per-scene Optimization?

I run this command to do Per-scene optimize from scratch:
bash dev_scripts/w_n360/lego.sh

In the paper, the speed should be 2min / 1K iters. However, on my single 2080Ti, it takes 100min / 1K iters. Anyone meets this problem?

for i in $(seq 1 $prob_freq $maximum_step)

for i in $(seq 1 $prob_freq $maximum_step)

What does the above code do？

Generic or per-scene NeRF?

After training on the dtu dataset, is it possible to also render all the test scenes in the dtu dataset without optimizing each scene? So pointnerf also seems to be a generic nerf like pixelnerf?
Then after per-scene optimization, can pointnerf get better performance than Nerf trained with multiple views?

Arguments are required --name

Hi everyone,
I have a problem when launching bash scripts like "bash dev_scripts/w_n360/chair.sh", the argument 'name' is not detected and the terminal returns me the error:
train_ft.py: error: the following arguments are required: --name
Did any of you had the same problem?

ValueError when generate video

Thank you for your work!
There is a value error when I test the per-scene optimized Point-NeRFs.
For example, run:
bash dev_scripts/w_n360/chair_test.sh

I checked the line 92 in visualizer.py and find the total_step is not a integer but a string.

Therefore, the filename cannot be generated and will bring this error because of mismatched variable type.
You can fix this bug by converting total_step to int.
Thank you for your time.

The process of training my own data

Thank you for your work and contribution.
I saw that you uploaded the code for COLMAP and it seems to only need the colmap_results/dense/fused.ply file.
I'm having trouble making my own dataset, I have a batch of images now, how do I make my transforms_{}.json. the camera_angle_x and rotation in it I don't know where to get from, the transform_matrix seems to be available from COLMAP. Do you have a script to generate these files please? I'm new at this.

what prempl does？

Thanks for your great work!
I would like to know what premlp(https://github.com/Xharlie/pointnerf/blob/master/models/mvs/mvs_points_model.py line22) does, just turn 63-dimensional input into 32-dimensional? How is the weight of this part set?

about training precedure cost?

Thanks for your great work! I wonder about the time cost, iteration numbers, and device when training on DTU dataset. Would you please tell me this important information?

pycuda error?

Hi @Xharlie @zexxu ,

Could you help me about the following error?

Do you evaluate your method on a mixture of training and testing images of ScanNet?

I am confused about the training and testing split. Please see

pointnerf/data/scannet_ft_dataset.py

Line 302 in 9377094

else: # nsvf configuration

In the above code, when self.opt.test_num_step is 1, the whole self.all_id_list is returned as testing list. Since self.all_id_list contains all the frame id of the dataset, the codes evaluate methods on all frames of a scene, including the training frames.

Why '/ 4'?

Why '/ 4'?

pointnerf/data/nerf_synth360_ft_dataset.py

Line 393 in a614e1d

intrinsic[:2] = intrinsic[:2] / 4

Train 2D feature extraction and point representation

Hi @Xharlie and @zexxu ,

I do not understand about "point feature initial" part.
Does the model under "init" folder for all dataset? Or for different dataset, I need to train a VGG 2D feature extracting network firstly and seperately? If need, how to train for different dataset.

Can the model under "init" used by other dataset except for dtu? For example, scannet and kitti.

Best,
Yingjie

nerf_synth_config

Hi. Thanks for your great work. I am creating a data-set collecting by myself, I want to know how to write a fine-tune_pair.txt in the nerfsynth data-set. Looking forward to your reply.

Confidence of new point

In the training process(point growing), how to obtain the confidence of the newly added neural points, and how to optimize the confidence of these points.

which tool did you use to make the pipeline graphic image?

Hi @zexxu,

Which tool did you use to create this illustration?

hi how to use pointnerf on my own dataset?

thanks for your briliant project. I want to use pointnerf on my own dataset. should i retrain the mvsNet on my own dataset??

segmentation fault (core dumped)

hi, thanks for your great work.
what device do you use to train the model? I get segmentation fault (core dumped) when i train on p100 using bash dev_scripts/w_n360/chair.sh python 3.7.5 pytorch 1.8.1 pycuda 2021.1 torch_scatter2.0.8

What is the output?

Hello authors, thank you very much for your paper and for releasing code!

I am trying to run the script "bash dev_scripts/w_tt_ft/barn_test.sh". Is the final output a 3D model? I was confused.

This is the log file recorded at runtime.

out.txt

loss item

Thanks for sharing this work. i am a little confused about the loss item. i know that 'coarse_raycolor' is corresponding to L_render in paper, what is 'ray_masked_loss' and 'ray_miss_loss'?
And if i understand correctly, 'zero_one_loss_items' is the L_sparse in paper. but since we already set the neural points as input data and didn't load mvsnet model, can the parameters of mvsnet also be updated? (when i inspect the variable , the only model is "ray_marching") And you set the neural points to nn.parameter to update, any reasons behind it?

KeyError：'ray_mask'

I get an error when run train_ft.py. Why it happens?

Traceback (most recent call last):
File "train_ft.py", line 1086, in
main()
File "train_ft.py", line 869, in main
add_xyz, add_embedding, add_color, add_dir, add_conf = probe_hole(model, prob_dataset, Visualizer(prob_opt), prob_opt, None, test_steps=total_steps, opacity_thresh=opt.prob_thresh)
File "train_ft.py", line 497, in probe_hole
gt_image = torch.zeros((height * width, 3), dtype=torch.float32, device=prob_maps["ray_mask"].device)
KeyError: 'ray_mask'
end loading

scene101.sh and scene101_test.sh are completely the same

Hi! Great work. But I think there might be some mistake between the above two files. The codes inside them are entirely the same.

CUDA driver error

Thanks for your ncie work.
However, I have no idea with this bug，really need help.