weiyithu / nerfingmvs

[ICCV 2021 Oral] NerfingMVS: Guided Optimization of Neural Radiance Fields for Indoor Multi-view Stereo

NerfingMVS

Project Page | Paper | Video | Data

NerfingMVS: Guided Optimization of Neural Radiance Fields for Indoor Multi-view Stereo
Yi Wei, Shaohui Liu, Yongming Rao, Wang Zhao, Jiwen Lu, Jie Zhou
ICCV 2021 (Oral Presentation)

Installation

• Pull NerfingMVS repo.

  git clone --recursive [email protected]:weiyithu/NerfingMVS.git
  

• Install python packages with anaconda.

  conda create -n NerfingMVS python=3.7
  conda activate NerfingMVS
  conda install pytorch==1.7.1 torchvision==0.8.2 torchaudio==0.7.2 -c pytorch
  pip install -r requirements.txt
  

• We use COLMAP to calculate poses and sparse depths. However, original COLMAP does not have fusion mask for each view. Thus, we add masks to COLMAP and denote it as a submodule. Please follow https://colmap.github.io/install.html to install COLMAP in ./colmap folder (Note that do not cover colmap folder with the original version).

Usage

• Download 8 ScanNet scene data used in the paper here and put them under ./data folder. We also upload final results and checkpoints of each scene here.
• Run NerfingMVS

  sh run.sh $scene_name
  

  The whole procedure takes about 3.5 hours on one NVIDIA GeForce RTX 2080 GPU, including COLMAP, depth priors training, NeRF training, filtering and evaluation. COLMAP can be accelerated with multiple GPUs.You will get per-view depth maps in ./logs/$scene_name/filter. Note that these depth maps have been aligned with COLMAP poses. COLMAP results will be saved in ./data/$scene_name while others will be preserved in ./logs/$scene_name

Run on Your Own Data!

• Place your data with the following structure:

  NerfingMVS
  |───data
  |    |──────$scene_name
  |    |   |   train.txt
  |    |   |──────images
  |    |   |    |    001.jpg
  |    |   |    |    002.jpg
  |    |   |    |    ...
  |───configs
  |    $scene_name.txt
  |     ...
  

  train.txt contains names of all the images. Images can be renamed arbitrarily and '001.jpg' is just an example. You also need to imitate ScanNet scenes to create a config file in ./configs. Note that factor parameter controls the resolution of output depth maps. You also should adjust depth_N_iters, depth_H, depth_W in options.py accordingly.
• Run NerfingMVS without evaluation

  sh demo.sh $scene_name
  

  Since our work currently relies on COLMAP, the results are dependent on the quality of the acquired poses and sparse reconstruction from COLMAP.

Acknowledgement

Our code is based on the pytorch implementation of NeRF: NeRF-pytorch. We also refer to mannequin challenge.

Citation

If you find our work useful in your research, please consider citing:

@inproceedings{wei2021nerfingmvs,
  author    = {Wei, Yi and Liu, Shaohui and Rao, Yongming and Zhao, Wang and Lu, Jiwen and Zhou, Jie},
  title     = {NerfingMVS: Guided Optimization of Neural Radiance Fields for Indoor Multi-view Stereo},
  booktitle = {ICCV},
  year = {2021}
}

nerfingmvs's Issues

最终的视差结果是什么格式

请问一下，
在log/$scene/filter目录下生成了一些视差文件。这些视差文件是什么格式，怎么转成真实的视差。

Result degradation when training from one image

Hi author, I try to train your work from one image that you deleted from the original Nerf. I use the same code but render worse results than training from the random batch. I really want to know whether you've faced this situation and I can't explain why it happened...

why 'depth_N_rand' is default 4?

Hello sir, I have something to ask you. I want to know why the 'depth_N_rand' is default 4? and what it means? can it be any other value?

Why Sparse depth comes from MVS instead of SfM in Code?

Hi,
I found you used COLMAP's MVS pipline in your code to gennerate fused Point Cloud and then get the depth map. But you said you use COLMAP's SfM in your paper (the sparse depth in Fig.2 is also from the MVS instead of SfM). The sparse depth from SfM is actually very sparse. If you use COLMAP's MVS to reconsrtuct the point cloud, the depth from point cloud is already accurate (though a little sparse). So it confused me a lot. Why do you use nerf to generate depth? The Tab. 3 shows the depth from DepthNet is very accurate, the Nerf helps a little. I guess if you use a deeper DepthNet and a better loss you can get a better depth.

struct.error: unpack requires a buffer of 12 bytes

Hello,author！I want to ask you some questions about running code and reporting errors.I run it step by step according to the steps in your readme, but I'm running sh run.sh scene0000_01,after running for about 20minutes,an error was reoprted as follow:
Traceback (most recent call last):
File "run.py", line 19, in
depth_priors.train(args)
File "/home/ubuntu/ygx/NerfingMVS-main/src/depth_priors.py", line 70, in train
args.depth_H, args.depth_W)
File "/home/ubuntu/ygx/NerfingMVS-main/utils/io_utils.py", line 27, in load_colmap
ply_masks = read_ply_mask(ply_path)
File "/home/ubuntu/ygx/NerfingMVS-main/utils/colmap_utils.py", line 62, in read_ply_mask
load_point_vis(ply_vis_path, masks)
File "/home/ubuntu/ygx/NerfingMVS-main/utils/colmap_utils.py", line 41, in load_point_vis
idx, u, v = struct.unpack('<III', f.read(4 * 3))
struct.error: unpack requires a buffer of 12 bytes
I have also consulted the relevant information about error reporting, but it still can't be solved. I hope author can give some solutions.Thank you!

Results quality of NeRF

I downloaded the results you provided and found the results of NeRF do not match the ones shown in the paper. The NeRF results in the paper seem much worse. Could you please expain what is going on here? Thanks!

colmap make

Why would this problem happen? I used your colmap version(without git clone or checkout): cmake & make. Then it's wring during the process of making

[ 50%] Building CXX object src/CMakeFiles/colmap.dir/mvs/meshing.cc.o
[ 50%] Building CXX object src/CMakeFiles/colmap.dir/optim/bundle_adjustment.cc.o
[ 50%] Building CXX object src/CMakeFiles/colmap.dir/ui/undistortion_widget.cc.o
/home/lichao/Documents/NerfingMVS/colmap/src/estimators/pose.cc: In function ‘bool colmap::RefineAbsolutePose(const colmap::AbsolutePoseRefinementOptions&, const std::vector&, const std::vector<Eigen::Matrix<double, 2, 1>, std::allocator<Eigen::Matrix<double, 2, 1> > >&, const std::vector<Eigen::Matrix<double, 3, 1> >&, Eigen::Vector4d*, Eigen::Vector3d*, colmap::Camera*)’:
/home/lichao/Documents/NerfingMVS/colmap/src/estimators/pose.cc:242:20: error: ‘QuaternionParameterization’ in namespace ‘ceres’ does not name a type
242 | new ceres::QuaternionParameterization;
| ^~~~~~~~~~~~~~~~~~~~~~~~~~
/home/lichao/Documents/NerfingMVS/colmap/src/estimators/pose.cc:243:13: error: ‘class ceres::Problem’ has no member named ‘SetParameterization’; did you mean ‘SetParameterLowerBound’?
243 | problem.SetParameterization(qvec->data(), quaternion_parameterization);
| ^~~~~~~~~~~~~~~~~~~
| SetParameterLowerBound
/home/lichao/Documents/NerfingMVS/colmap/src/estimators/pose.cc:276:16: error: ‘SubsetParameterization’ is not a member of ‘ceres’; did you mean ‘LocalParameterization’?
276 | ceres::SubsetParameterization* camera_params_parameterization =
| ^~~~~~~~~~~~~~~~~~~~~~
| LocalParameterization
/home/lichao/Documents/NerfingMVS/colmap/src/estimators/pose.cc:276:40: error: ‘camera_params_parameterization’ was not declared in this scope
276 | ceres::SubsetParameterization* camera_params_parameterization =
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/home/lichao/Documents/NerfingMVS/colmap/src/estimators/pose.cc:277:17: error: expected type-specifier
277 | new ceres::SubsetParameterization(
| ^~~~~
/home/lichao/Documents/NerfingMVS/colmap/src/estimators/pose.cc:279:17: error: ‘class ceres::Problem’ has no member named ‘SetParameterization’; did you mean ‘SetParameterLowerBound’?
279 | problem.SetParameterization(camera->ParamsData(),
| ^~~~~~~~~~~~~~~~~~~
| SetParameterLowerBound
/home/lichao/Documents/NerfingMVS/colmap/src/estimators/pose.cc: In function ‘bool colmap::RefineRelativePose(const ceres::Solver::Options&, const std::vector<Eigen::Matrix<double, 2, 1>, std::allocator<Eigen::Matrix<double, 2, 1> > >&, const std::vector<Eigen::Matrix<double, 2, 1>, std::allocator<Eigen::Matrix<double, 2, 1> > >&, Eigen::Vector4d*, Eigen::Vector3d*)’:
/home/lichao/Documents/NerfingMVS/colmap/src/estimators/pose.cc:333:18: error: ‘QuaternionParameterization’ in namespace ‘ceres’ does not name a type
333 | new ceres::QuaternionParameterization;
| ^~~~~~~~~~~~~~~~~~~~~~~~~~
/home/lichao/Documents/NerfingMVS/colmap/src/estimators/pose.cc:334:11: error: ‘class ceres::Problem’ has no member named ‘SetParameterization’; did you mean ‘SetParameterLowerBound’?
334 | problem.SetParameterization(qvec->data(), quaternion_parameterization);
| ^~~~~~~~~~~~~~~~~~~
| SetParameterLowerBound
/home/lichao/Documents/NerfingMVS/colmap/src/estimators/pose.cc:336:10: error: ‘HomogeneousVectorParameterization’ is not a member of ‘ceres’
336 | ceres::HomogeneousVectorParameterization* homogeneous_parameterization =
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/home/lichao/Documents/NerfingMVS/colmap/src/estimators/pose.cc:336:45: error: ‘homogeneous_parameterization’ was not declared in this scope; did you mean ‘quaternion_parameterization’?
336 | ceres::HomogeneousVectorParameterization* homogeneous_parameterization =
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~
| quaternion_parameterization
/home/lichao/Documents/NerfingMVS/colmap/src/estimators/pose.cc:337:11: error: expected type-specifier
337 | new ceres::HomogeneousVectorParameterization(3);
| ^~~~~
/home/lichao/Documents/NerfingMVS/colmap/src/estimators/pose.cc:338:11: error: ‘class ceres::Problem’ has no member named ‘SetParameterization’; did you mean ‘SetParameterLowerBound’?
338 | problem.SetParameterization(tvec->data(), homogeneous_parameterization);
| ^~~~~~~~~~~~~~~~~~~
| SetParameterLowerBound
/home/lichao/Documents/NerfingMVS/colmap/src/optim/bundle_adjustment.cc: In member function ‘void colmap::BundleAdjuster::AddImageToProblem(colmap::image_t, colmap::Reconstruction*, ceres::LossFunction*)’:
/home/lichao/Documents/NerfingMVS/colmap/src/optim/bundle_adjustment.cc:420:22: error: ‘QuaternionParameterization’ in namespace ‘ceres’ does not name a type
420 | new ceres::QuaternionParameterization;
| ^~~~~~~~~~~~~~~~~~~~~~~~~~
/home/lichao/Documents/NerfingMVS/colmap/src/optim/bundle_adjustment.cc:421:17: error: ‘class ceres::Problem’ has no member named ‘SetParameterization’; did you mean ‘SetParameterLowerBound’?
421 | problem_->SetParameterization(qvec_data, quaternion_parameterization);
| ^~~~~~~~~~~~~~~~~~~
| SetParameterLowerBound
/home/lichao/Documents/NerfingMVS/colmap/src/optim/bundle_adjustment.cc:425:16: error: ‘SubsetParameterization’ is not a member of ‘ceres’; did you mean ‘LocalParameterization’?
425 | ceres::SubsetParameterization* tvec_parameterization =
| ^~~~~~~~~~~~~~~~~~~~~~
| LocalParameterization
/home/lichao/Documents/NerfingMVS/colmap/src/optim/bundle_adjustment.cc:425:40: error: ‘tvec_parameterization’ was not declared in this scope; did you mean ‘quaternion_parameterization’?
425 | ceres::SubsetParameterization* tvec_parameterization =
| ^~~~~~~~~~~~~~~~~~~~~
| quaternion_parameterization
/home/lichao/Documents/NerfingMVS/colmap/src/optim/bundle_adjustment.cc:426:17: error: expected type-specifier
426 | new ceres::SubsetParameterization(3, constant_tvec_idxs);
| ^~~~~
/home/lichao/Documents/NerfingMVS/colmap/src/optim/bundle_adjustment.cc:427:19: error: ‘class ceres::Problem’ has no member named ‘SetParameterization’; did you mean ‘SetParameterLowerBound’?
427 | problem_->SetParameterization(tvec_data, tvec_parameterization);
| ^~~~~~~~~~~~~~~~~~~
| SetParameterLowerBound
/home/lichao/Documents/NerfingMVS/colmap/src/optim/bundle_adjustment.cc:423:33: warning: unused variable ‘constant_tvec_idxs’ [-Wunused-variable]
423 | const std::vector& constant_tvec_idxs =
| ^~~~~~~~~~~~~~~~~~
/home/lichao/Documents/NerfingMVS/colmap/src/optim/bundle_adjustment.cc: In member function ‘void colmap::BundleAdjuster::ParameterizeCameras(colmap::Reconstruction*)’:
/home/lichao/Documents/NerfingMVS/colmap/src/optim/bundle_adjustment.cc:514:16: error: ‘SubsetParameterization’ is not a member of ‘ceres’; did you mean ‘LocalParameterization’?
514 | ceres::SubsetParameterization* camera_params_parameterization =
| ^~~~~~~~~~~~~~~~~~~~~~
| LocalParameterization
/home/lichao/Documents/NerfingMVS/colmap/src/optim/bundle_adjustment.cc:514:40: error: ‘camera_params_parameterization’ was not declared in this scope
514 | ceres::SubsetParameterization* camera_params_parameterization =
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/home/lichao/Documents/NerfingMVS/colmap/src/optim/bundle_adjustment.cc:515:17: error: expected type-specifier
515 | new ceres::SubsetParameterization(
| ^~~~~
/home/lichao/Documents/NerfingMVS/colmap/src/optim/bundle_adjustment.cc:517:19: error: ‘class ceres::Problem’ has no member named ‘SetParameterization’; did you mean ‘SetParameterLowerBound’?
517 | problem_->SetParameterization(camera.ParamsData(),
| ^~~~~~~~~~~~~~~~~~~
| SetParameterLowerBound
/home/lichao/Documents/NerfingMVS/colmap/src/optim/bundle_adjustment.cc: In member function ‘void colmap::RigBundleAdjuster::AddImageToProblem(colmap::image_t, colmap::Reconstruction*, std::vectorcolmap::CameraRig, ceres::LossFunction)’:
/home/lichao/Documents/NerfingMVS/colmap/src/optim/bundle_adjustment.cc:1061:14: error: ‘SubsetParameterization’ is not a member of ‘ceres’; did you mean ‘LocalParameterization’?
1061 | ceres::SubsetParameterization* tvec_parameterization =
| ^~~~~~~~~~~~~~~~~~~~~~
| LocalParameterization
/home/lichao/Documents/NerfingMVS/colmap/src/optim/bundle_adjustment.cc:1061:38: error: ‘tvec_parameterization’ was not declared in this scope
1061 | ceres::SubsetParameterization* tvec_parameterization =
| ^~~~~~~~~~~~~~~~~~~~~
/home/lichao/Documents/NerfingMVS/colmap/src/optim/bundle_adjustment.cc:1062:15: error: expected type-specifier
1062 | new ceres::SubsetParameterization(3, constant_tvec_idxs);
| ^~~~~
/home/lichao/Documents/NerfingMVS/colmap/src/optim/bundle_adjustment.cc:1063:17: error: ‘class ceres::Problem’ has no member named ‘SetParameterization’; did you mean ‘SetParameterLowerBound’?
1063 | problem_->SetParameterization(tvec_data, tvec_parameterization);
| ^~~~~~~~~~~~~~~~~~~
| SetParameterLowerBound
/home/lichao/Documents/NerfingMVS/colmap/src/optim/bundle_adjustment.cc:1059:31: warning: unused variable ‘constant_tvec_idxs’ [-Wunused-variable]
1059 | const std::vector& constant_tvec_idxs =
| ^~~~~~~~~~~~~~~~~~
/home/lichao/Documents/NerfingMVS/colmap/src/optim/bundle_adjustment.cc: In member function ‘void colmap::RigBundleAdjuster::ParameterizeCameraRigs(colmap::Reconstruction*)’:
/home/lichao/Documents/NerfingMVS/colmap/src/optim/bundle_adjustment.cc:1147:20: error: ‘QuaternionParameterization’ in namespace ‘ceres’ does not name a type
1147 | new ceres::QuaternionParameterization;
| ^~~~~~~~~~~~~~~~~~~~~~~~~~
/home/lichao/Documents/NerfingMVS/colmap/src/optim/bundle_adjustment.cc:1148:15: error: ‘class ceres::Problem’ has no member named ‘SetParameterization’; did you mean ‘SetParameterLowerBound’?
1148 | problem_->SetParameterization(qvec_data, quaternion_parameterization);
| ^~~~~~~~~~~~~~~~~~~
| SetParameterLowerBound
make[2]: *** [src/CMakeFiles/colmap.dir/build.make:661: src/CMakeFiles/colmap.dir/estimators/pose.cc.o] Error 1
make[2]: *** Waiting for unfinished jobs....
make[2]: *** [src/CMakeFiles/colmap.dir/build.make:882: src/CMakeFiles/colmap.dir/optim/bundle_adjustment.cc.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:858: src/CMakeFiles/colmap.dir/all] Error 2
make: *** [Makefile:149: all] Error 2

I hope to receive your reply! THX

Generate depth image ?

Hello , Thanks for your great work.

However ,I don't know how to generate depth image for my own data.Have you provided the code to generate depth image? How do you generate the depth map, render from the mesh.ply in the scannet ?

Project dependencies may have API risk issues

Hi, In NerfingMVS, inappropriate dependency versioning constraints can cause risks.

Below are the dependencies and version constraints that the project is using

lpips==0.1.3
imageio
opencv-python
scikit-image==0.15.0
tqdm
wget
configargparse
tensorboard
h5py

The version constraint == will introduce the risk of dependency conflicts because the scope of dependencies is too strict.
The version constraint No Upper Bound and * will introduce the risk of the missing API Error because the latest version of the dependencies may remove some APIs.

After further analysis, in this project,
The version constraint of dependency imageio can be changed to >=1.1-linux32,<=2.19.3.
The version constraint of dependency tqdm can be changed to >=4.36.0,<=4.64.0.
The version constraint of dependency configargparse can be changed to >=0.9.3,<=1.5.3.
The version constraint of dependency h5py can be changed to >=2.5.0,<=3.7.0.

The above modification suggestions can reduce the dependency conflicts as much as possible,
and introduce the latest version as much as possible without calling Error in the projects.

The invocation of the current project includes all the following methods.

The calling methods from the imageio

imageio.imwrite
imageio.imread

The calling methods from the tqdm

tqdm.tqdm.write
tqdm.tqdm
tqdm.trange

The calling methods from the configargparse

configargparse.ArgumentParser

The calling methods from the h5py

h5py.File.close
h5py.File

The calling methods from the all methods

bistochastize
targets.cuda
data_term.item
numpy.genfromtxt
self.blurs.append
hourglass.HourglassModel
self.slice
self.GradientLoss
numpy.ndarray.max
pix_coords.permute.to
mask.sum
numpy.linalg.norm
Image
torch.cat
rgb.cpu.numpy
pred_confidence.data.unsqueeze
loss_list.torch.stack.mean
img_grad_y.abs
numpy.isfinite
scipy.sparse.diags
file.write
poses.np.moveaxis.astype
i.input_depth.data.cpu.numpy
self.S.dot
k.ret.torch.isinf.any
torch.clamp
super.__init__
i.d_gt.cpu
get_valid_idx
torch.Tensor.expand
h5py.File.create_dataset
torch.rand
sc_inv_full.item
i.prediction_d.data.cpu.numpy
torch.nn.Identity
i.pred_d.cpu.numpy
poses.torch.from_numpy.to.clone
list
numpy.array
torch.Tensor
torch.ones_like
eo.embed
i.d_gt.cpu.numpy
self.list.append
torch.pow
lpips_list.append
masks.torch.from_numpy.to
argparse.ArgumentParser
rays_o.torch.reshape.float.clone
numpy.argsort
self.netG.parameters
img2mse
read_cameras_text
render
i.full_flow.data.cpu.numpy
zip
time.time
torch.nn.DataParallel.load_state_dict
self.register_buffer
models.depth_priors.mannequin_challenge_model.MannequinChallengeModel.train
subprocess.check_output
torch.nn.init.xavier_normal_
minify
self.blur
input.view
models.depth_priors.mannequin_challenge_model.MannequinChallengeModel
h5py.File
skimage.io.imsave
lpips_metric.cpu
load_depths
k.all_ret.append
ret.append
rays_d.torch.reshape.float.clone
bds.np.moveaxis.astype.max
read_images_text
rgb_evaluation
rays_rgb.torch.Tensor.to.astype
network.cpu
rays_rgb.torch.Tensor.to
numpy.eye
disp.cpu
sigma_spatial.Iy.astype
prediction_d.data.cpu
render_path_spiral
torch.optim.lr_scheduler.StepLR
cv2.imread
img_grad_x.abs.mean
invK.torch.unsqueeze.repeat
lpips_list.np.concatenate.mean
torch.cumprod
net_.parameters
numpy.flatnonzero
numpy.mean
visualize_depth
X.copy
numpy.array.reshape
rays_o.torch.reshape.float
torch.autograd.set_detect_anomaly
Channels4
self.seq
Z.reshape.astype
numpy.expand_dims
self.Channels3.super.__init__
trans.append
f.read
scipy.sparse.csr_matrix.dot
torch.median
input_.cuda
numpy.array.append
compute_depth_loss.backward
numpy.matmul
numpy.arange
p_fn
vars.items
tqdm.tqdm.write
models.depth_priors.mannequin_challenge_model.MannequinChallengeModel.forward
coord.reshape
load_rgbs_np
self.new_model
i.self.views_linears
i.full_flow.data.cpu
input_images.data.cpu
torch.nn.init.kaiming_normal_
x.squeeze.squeeze
torch.nn.Sequential
i.gt_mask.cpu.numpy
depths.cpu.numpy
prediction_d.data.cpu.torch.exp.unsqueeze
self.loss_joint_var.backward
targets.unsqueeze
f.strip.replace
numpy.argmin
c.t.bs_params.grid.BilateralSolver.solve.reshape
numpy.percentile
models.depth_priors.mannequin_challenge_model.MannequinChallengeModel.eval
self.input.cuda
ssim_list.append
numpy.sqrt
render_rays
numpy.searchsorted
numpy.dstack
depth_grad_yx.abs.mean
img.dim
numpy.tensordot
im.qvec2rotmat
compute_errors
depth_model.model.netG.state_dict
i.input_imgs.cpu.numpy
numpy.load.transpose
numpy.argmax
torch.FloatTensor.unsqueeze
self.backward_G
image_name.replace.split
optimizer_depth.zero_grad
input.size
img_lap.mean.exp.squeeze
inception
depths.torch.from_numpy.to
numpy.shape
main
optimizer_depth.step
float
mse2psnr
self.model.netG.forward
run_network
torchvision.transforms.ToTensor
lpips.LPIPS
batch.depth_model.forward.cpu
numpy.load.astype
torch.sigmoid.cpu
torch.nn.BatchNorm2d
self.create_embedding_fn
f.endswith
predicts.transpose.torch.from_numpy.type
numpy.sum
bds.np.moveaxis.astype
grid.splat
self.rgb_linear
grad_term.item
torch.nn.Sigmoid
numpy.column_stack
torch.matmul.unsqueeze
im.astype
any
W.reshape.astype
perm.append
numpy.linalg.eigh
depth_confidences.torch.reshape.float
numpy.logical_not
tqdm.trange
torch.transpose
numpy.concatenate.append
poses.astype.sum
masks.append
self.criterion_joint
set
src.depth_priors.train
metadata.unsqueeze.cuda
torch.nn.functional.relu
self.writer.add_image
torch.nn.Linear
mannequin_challenge.options.train_options.TrainOptions
LaplacianLayer
i.input_imgs.cpu
numpy.newaxis.pts_arr.transpose
numpy.stack.split
depth.view
self.HourglassModel.super.__init__
struct.unpack
w.reshape.reshape
self.write_summary
numpy.max
self.convs.append
numpy.maximum.mean
self.grid.slice
torch.log
self.laplacian_func
numpy.moveaxis
self.Data_Loss
network.cpu.state_dict
cal_depth_confidences
int
vec_from_R
torch.abs
ndc_rays
numpy.linalg.inv.transpose
skimage.measure.compare_ssim
depth_priors.torch.reshape.float
torch.utils.tensorboard.SummaryWriter
self.criterion_joint.compute_rmse_error
src.initialize.main
torch.optim.Adam.step
torch.stack.cpu
skimage.transform.resize
torch.FloatTensor.gts.transpose.torch.from_numpy.type.cuda
self.model.switch_to_train
depth_grad_yx.abs
image_name.split
gts.astype.max
open
sigma_chroma.im_yuv.astype
valid_z.max
self.LaplacianLayer.super.__init__
W.reshape
embed_fns.append
torch.nn.init.normal_
disp.cpu.numpy
poses.torch.Tensor.to
mse2psnr.item
self.Dn.dot
i.mvs_depth.data.cpu.numpy
solve_image_ldl3
img_name.split
numpy.unique
read_model
torch.inverse
sigma_luma.im_yuv.astype
fid.seek
self.opt.gpu_ids.split
self.Channels4.super.__init__
i.i.cam_points.repeat
pred_confidence.data.unsqueeze.repeat
configargparse.ArgumentParser
Project3D
viewdirs.torch.reshape.float.expand
torch.nn.init.orthogonal_
multiprocessing.Pool
temp.cpu.numpy
self.estimate_depth
cv2.imwrite
images.reshape.cuda
scipy.sparse.linalg.cg
numpy.load.reshape
self.targets.cuda
format
fid.readline
i.targets.split
poses_avg
raw2outputs
torch.stack
torch.unsqueeze
imgs.np.moveaxis.astype.split
exit
EPSILON.mask.data.pred_d.data.mask.data.gt_d.data.torch.median.item
i.pred_confidence.data.cpu
depth_grad_y2.abs
torch.norm
functools.partial
A.diagonal
self.load_network
os.listdir
i_train.len.np.arange.repeat
load_gt_depths
fid.readline.split
i.t.t
src.run_nerf.train
depth_model.model.netG.load_state_dict
i.human_mask.data.cpu.numpy
self.Data_Human_Loss
torch.nn.Sequential.children
meshgrid.np.stack.astype
torch.FloatTensor
self.criterion_joint.compute_si_rmse
os.path.exists
img_lap.mean.exp.view
numpy.random.shuffle
self.uncertainty_layer
fid.read
self.criterion_joint.get_loss_var
self.netG.train
self.optimizer_G.zero_grad
numpy.reshape
rays_d.torch.reshape.float
numpy.logical_or
img_grad_y.abs.mean
input.unsqueeze.unsqueeze
self.model.netG.parameters
init_weights
numpy.cross
predicts.gts.mean.mean.mean
h5py.File.close
zipfile.ZipFile
A_i.A_i.np.transpose.mean
b_i.mean
torch.nn.functional.conv2d.abs
self.netG.eval
Channels1
depth.cpu
bds.np.moveaxis.astype.min
numpy.uint8
torch.randn
compute_depth_loss
img_lap.mean.exp.mean
self.model.netG.state_dict
numpy.linspace
sc_inv_human.item
torch.sum.item
mannequin_challenge.options.train_options.TrainOptions.initialize
numpy.random.rand
_load_data
targets.cuda.autograd.Variable.unsqueeze
gts.astype.transpose
gts.astype.astype
networks.get_scheduler
load_point_vis
self.save_network
network_query_fn
cal_depth_confidences.astype
BilateralSolver
batch.depth_model.forward.cpu.numpy
get_rays_np
multiprocessing.Pool.close
self.splat
torch.autograd.Variable
image_name.replace.replace
prediction_log_d.squeeze
H.focal.W.focal.torch.FloatTensor.to
cv2.resize.astype
img_lap.mean.exp
os.path.join
sc_inv_env.item
recenter_poses
numpy.isnan
self.HourglassVariant.super.__init__
torch.matmul
z_vals.expand.expand
weight_y.depth_grad_y.abs.weight_x.depth_grad_x.abs.mean
read_images_binary.items
pil_loader
torch.randperm
SuppressedStdout
os.path.isfile
self.parser.parse_args
torch.max
i.i.T.repeat
render_path
torch.ones
f.w.h.np.array.reshape
numpy.zeros
images.astype.astype
depth_confidences.append
utils.colmap_read_model.read_cameras_binary
torch.abs.sum
self.scheduler.step
self.feature_linear
local_path.rstrip.rstrip
torchvision.utils.make_grid
tran_dis.sum.sum
line.strip.strip
self.model.prediction_d.squeeze
self.netG.forward
torch.isinf
torch.load
network.cuda
torch.meshgrid
self.grid.splat
wget.download
torch.split
torch.FloatTensor.view.cuda
numpy.ndarray.min
gt_depths_valid.append
os.getcwd
saved_imgs.astype.astype
self.alpha_linear
netchunk.fn.batchify
os.remove
Z.reshape
networks.print_network
k.ret.torch.isnan.any
torch.nn.functional.grid_sample
depth_grad_y2.abs.mean
utils.colmap_read_model.read_points3d_binary
predicts.astype.astype
sh.np.array.reshape
dict
torch.sum
pix_coords.permute.permute
torch.matmul.view
utils.colmap_read_model.read_images_binary
self.inception.super.__init__
numpy.abs
Channels3
torch.autograd.Variable.size
angles.append
i.input_depth.data.cpu
depths.append
self.pred_confidence.squeeze
images.torch.Tensor.to
super
read_images_binary
sorted
f.readlines
print
numpy.sin
numpy.save
depth_grad_x.abs
numpy.transpose
min_line_dist
pred_confidence.squeeze.squeeze
ValueError
Project3D_depth
configargparse.ArgumentParser.add_argument
gradient
mkdir
numpy.log
qvec2rotmat
angle_dis.sum.sum
numpy.linalg.inv
render_poses.np.array.astype
load_rgbs.clone
get_embedder
isinstance
torch.device
poses_tensor.shape.bottom.repeat.to.repeat
sys.stdout.close
bs_params.grid.BilateralSolver.solve
errors.np.array.mean.tolist
self.initialize
numpy.arcsin
im.tvec.reshape
numpy.log10
numpy.meshgrid
depth_grad_y.abs
options.config_parser.parse_args
util.util.mkdirs
numpy.median
self.forward
i.pred_confidence.data.cpu.numpy
numpy.dstack.reshape
self.criterion_joint.compute_l1_rel_error
metadata.unsqueeze
f.writelines
batchify_rays
base_options.BaseOptions.initialize
param.numel
numpy.tile
i.imgs_down.astype
self.JointLoss.super.__init__
torch.from_numpy.view
torch.sqrt
rot.transpose
numpy.square
i.image_list.split
_minify
spherify_poses
prediction_d.data.cpu.torch.exp.unsqueeze.repeat
prediction_d.data.cpu.torch.exp.unsqueeze.repeat.numpy
read_points3D_text
i.mvs_depth.data.cpu
i.input_confidence.data.cpu.numpy
array.np.transpose.squeeze
self.pred_layer
numpy.fromfile
vars
torch.nn.AvgPool2d
input.dim
join
self._compute_factorization
load_llff.load_llff_data
render_poses.np.array.astype.append
i.pred_d.cpu
depth_confidence.mean.cpu.numpy
torch.isnan
self.grid.blur
torch.no_grad
depth_confidence.mean.cpu
Channels2
torch.mean
get_model_from_url
NeRF
numpy.empty_like
f.extractall
torch.nn.DataParallel.to
FixedMcModel
self.optimizer_G.step
torch.FloatTensor.view
create_nerf
poses_arr.reshape.transpose
numpy.ones_like
imgs.np.moveaxis.astype
numpy.stack
sys.path.append
K.torch.unsqueeze.repeat
cv2.applyColorMap
ssim_list.np.array.mean
args.config.open.read
numpy.maximum
networks.JointLoss
numpy.load
sm_term.item
i.prediction_d.data.cpu
sc_inv_intra.item
imread
pred_depths_valid.append
predicts.gts.mean.mean
numpy.ones
self.parser.add_argument
self.laplacian_func.mean
depth_grad_xy.abs
rgbs.append
valid_z.mean
predicts.gts.mean
imageio.imwrite
numpy.clip
errors.np.array.mean
torch.optim.Adam.load_state_dict
downsample
line.strip.split
BackprojectDepth
total_loss.item
max
read_array
i.pred_d.data.cpu
input.unsqueeze
viewdirs.torch.reshape.float
depth_grad_xy.abs.mean
numpy.linalg.lstsq
torch.optim.Adam.zero_grad
images.reshape.reshape
pix_coords_ref.i_train.depths.unsqueeze.F.grid_sample.squeeze
BilateralGrid
fn
torch.nn.UpsamplingBilinear2d
i.poses.ravel
numpy.squeeze
self.S.T.dot
loss.backward
torch.nn.init.constant_
numpy.cos
predicts.astype.transpose
filt
os.chdir
numpy.broadcast_to
p34_to_44
self.compute_image_aware_1st_smoothness_cost
depth_evaluation
far_bound.near_bound.depth_confidences.torch.clamp.depth_priors.unsqueeze
utils.pose_utils.gen_poses
i.self.pts_linears
render_kwargs_test.update
compute_depth_loss.item
numpy.ceil
loss.item
rgb2yuv
self.LaplacianSmoothnessLoss
array.reshape.reshape
Embedder
pred_d.unsqueeze
torch.reshape
sys.exit
i.poses.copy
CameraModel
torch.save
map
PIL.Image.open
embed_fn
torch.abs.topk
hasattr
torch.cuda.is_available
classname.find
src.filter.main
collections.namedtuple
str
read_points3d_binary
self._hash_coords
colmap_depths.shape.np.ones.astype
x.np.clip.astype
depth_confidences.torch.from_numpy.to
torch.FloatTensor.predicts.transpose.torch.from_numpy.type.cuda
args.use_viewdirs.input_ch_views.skips.output_ch.input_ch.args.netwidth.args.netdepth.NeRF.to
self.opt.gpu_ids.append
len
i.pred_d.data.cpu.numpy
read_next_bytes.decode
depth_grad_x2.abs.mean
os.makedirs
Camera
numpy.random.seed
torch.optim.lr_scheduler.ReduceLROnPlateau
read_next_bytes
errors.append
load_img_list
tuple
pix_coords.permute.view
mse.np.log10.mean
f.strip
mask_0.size
tqdm.tqdm
torch.linspace
poses.torch.from_numpy.to
sigma_spatial.Ix.astype
optimizer_depth.state_dict
torch.set_default_tensor_type
batchify
self.prediction_d.squeeze
self.model.switch_to_eval
act_fn
depth.cpu.numpy
torch.utils.tensorboard.SummaryWriter.add_scalar
utils.colmap_read_model.read_cameras_binary.keys
load_rgbs
poses.astype.mean
i_train.depths.unsqueeze
to_tensor
to8b
numpy.sqrt.mean
torch.mul
depth_model
NotImplementedError
lpips_metric
loss_list.append
parser.parser.parse_args
save_poses
models.depth_priors.mannequin_challenge_model.MannequinChallengeModel.parameters
torch.sigmoid
read_ply_mask
create_depth_model
depth_grad_x2.abs
log_pred.log_target.sum
torchvision.transforms.Resize
get_rays
options.config_parser
poses_tensor.shape.bottom.repeat.to
self.NeRF.super.__init__
poses.astype.astype
self.output_linear
render_kwargs_train.update
calculate_coords
align_scales
args.use_viewdirs.input_ch_views.skips.output_ch.input_ch.args.netwidth.args.netdepth.NeRF.to.load_state_dict
numpy.concatenate
normalize
conv
imageio.imread
i.gt_mask.cpu
line.strip
f_list.readlines
self.list
numpy.dot
grid.blur
torch.multiprocessing.set_start_method
load_colmap_data
c2w.expand
center.poses.mean
torch.stack.append
self.targets.cuda.autograd.Variable.unsqueeze
torch.optim.lr_scheduler.LambdaLR
enumerate
planar_filter
i.human_mask.data.cpu
raw2alpha
i.input_confidence.data.cpu
range
embeddirs_fn
torch.exp.squeeze
torch.nn.DataParallel
img.convert
scipy.sparse.csr_matrix
torch.nn.functional.conv2d
args.use_viewdirs.input_ch_views.skips.output_ch.input_ch.args.netwidth.args.netdepth.NeRF.to.parameters
viewmatrix
render_kwargs_train.state_dict
os.path.dirname
img_grad_x.abs
torch.nn.Conv2d
train
src.evaluation.main
Point3D
j.t.t
multiprocessing.Pool.join
time.time.time
self.writer.add_scalar
cv2.resize
read_cameras_binary
cal_neighbor_idx
opt_file.write
getattr
torch.from_numpy
torch.nn.ModuleList
i_test.len.np.arange.repeat
torch.optim.Adam.state_dict
torch.nn.DataParallel.apply
gt_rgbs.pred_rgbs.np.abs.mean
self.Channels1.super.__init__
valid_z.min
torch.optim.Adam
self.model.prediction_d.reshape
gts.transpose.torch.from_numpy.type
depth_priors.torch.from_numpy.to
resize
pred_d.unsqueeze.repeat
sc_inv_inter.item
numpy.stack.append
self.Channels2.super.__init__
torch.nn.ReLU
depth_confidence.mean.cpu.numpy.mean
load_colmap
multiprocessing.Pool.map_async
targets.unsqueeze.repeat
torch.exp

@developer
Could please help me check this issue?
May I pull a request to fix it?
Thank you very much.

Error maps calculation

Hi,

I had a few questions about how the error map is calculated:
from here it is understood that the formula to calculate error maps is:
D_dash - D_projected / D_projected + 10e-7, and from your paper, D_dash refers to the depth priors obtained from the monocular depth network that is indexed with the 2D pixel coordinates, and D_projected refers to the depth maps obtained from projecting from eg: view-0 to view-1,2,3,4..etc

I hope this is correct? because my questions are based on the assumption that the above is what is implied by the paper code, if so I will continue with my questions :)

Bad results on LLFF dataset

Hi, I tried to test NerfingMVS on LLFF dataset but get bad results. For example, on scene horns in LLFF dataset, the orignal NeRF gets PSNR=28.85 while NerfingMVS only gets PSNR=19.41. One rendered test image looks like this:

My config file for this scene is:

expname = horns_test
basedir = ./logs
datadir = ./raw_data/horns

factor = 8

N_rand = 1024
N_samples = 64

use_viewdirs = True
raw_noise_std = 1e0

depth_H = 384
depth_W = 512

May I ask for your results on this scene if you have tested it before? I am curious why the performance looks so bad on this scene.

Thanks!

Colab notebook?

Thanks for sharing this fantastic work. Would you consider releasing a Google Colab notebook for less technically inclined people like myself to to try this out? Thanks in advance!

Is there a way to generate the mesh of the trained scene?

I'm wondering if there's a option or script in your code to generate the 3D mesh of the trained scene? There's such option in the nerf-pytorch, but your code seems to be based on a older version of nerf-pytorch. And it will be great if you can also explain how to generate images from a specific view, like from an unseen direction. Many thanks

Rendering novel views

Hi! Thank you for your work!
It was said in the paper, that it's possible to render novel views by using adapted depth priors from the nearest view. However, nothing was said about how to deal with missed regions, in case we adapt these predicted priors (I guess, here we need to warp them). I also didn't manage to find any code doing that. Could you please clarify this point?

Scannet images used for training

Thank you for your excellent work and for sharing the code!

To compare with your results, can you please share which images you used for training and testing from the Scannet sequences you evaluated on?

Thanks!

Issue about install colmap

Hi,
I am trying to reproduce your work.
However, I face the issue about install col map.

I try to install from official web, conda and https://github.com/B1ueber2y/colmap.git.
They all fail to solve the problem. Would you mind to give me some suggestion?

无法渲染视频

作者您好，非常感谢你的工作！我在你的项目工程里面没有看到关于渲染120幅新颖视图的代码，render函数需要 depth_prior[i] 和 depth_confidence[i]，但这两个参数是基于训练视图的，因此无法渲染任意视角下的新颖图像，从而渲染视频。请问博主后面解决了这个问题了吗？或者是因为我在代码中忽略了哪些部分造成的误解吗？希望得到博主解惑，谢谢！

the correct camera poses for current points cannot be accessed

Great work ! But when I use my own dataset, I got

ERROR: the correct camera poses for current points cannot be accessed

And then I cannot get the "pose_bound.py".
So what should I do ?

The results do not agree with those in the paper

Hi, thank you for your excellent work!
When I try to reproduce the experiments in the paper, I can't get the same score as in the paper for 8 scenes.
Like scene0553, the rgb score is :

While I can only get the score below after 200001 iterations :

In my experiment, the hyperparameter Setting is the same as the code in the project:

May I ask if you can inform me the Hyperparameter Setting of the experiment with the score in the paper, as we are interested in reproducing the excellent experimental results! Thank you very much!

error with own data

Hi,
Thank you for your work. Everything seems to work well except if I use my own data.
This error happens:

Traceback (most recent call last):
File "run.py", line 19, in
depth_priors.train(args)
File "/mnt/work/Dev/Sources/mesh_refinement/NerfingMVS-main/src/depth_priors.py", line 86, in train
loss = compute_depth_loss(depth_pred, depth_gt, mask_gt)
File "/mnt/work/Dev/Sources/mesh_refinement/NerfingMVS-main/utils/depth_priors_utils.py", line 6, in compute_depth_loss
log_pred = torch.log(pred[mask])
IndexError: too many indices for tensor of dimension 1

not sure what's happening. Do you have any idea?

thank you

struct .error: unpack requires a buffer of 12 bytes

I met some errors when I had a try to run this code. Please help me!

I'm running Ubuntu 18.04.5 LTS
Python 3.7
Pytorch 1.9
Cuda 11.1

Depth stats 0.19367019493628088 315.65333461743836 61.20548491109312
Done with imgs2poses
Depths prior training begins !
Found cache /home/xxx/code/NerfingMVS/models/depth_priors/mc.pth
/home/xxx/miniconda2/envs/py37/lib/python3.7/site-packages/torchvision/transforms/transforms.py:281: UserWarning: Argument interpolation should be of type InterpolationMode instead of int. Please, use InterpolationMode enum.
  "Argument interpolation should be of type InterpolationMode instead of int. "
point number: 3922949
Traceback (most recent call last):
  File "run.py", line 19, in <module>
    depth_priors.train(args)
  File "/home/xxx/code/NerfingMVS/src/depth_priors.py", line 70, in train
    args.depth_H, args.depth_W)
  File "/home/xxx/code/NerfingMVS/utils/io_utils.py", line 27, in load_colmap
    ply_masks = read_ply_mask(ply_path)
  File "/home/xxx/code/NerfingMVS/utils/colmap_utils.py", line 52, in read_ply_mask
    load_point_vis(ply_vis_path, masks)
  File "/home/xxx/code/NerfingMVS/utils/colmap_utils.py", line 36, in load_point_vis
    idx, u, v = struct.unpack('<III', f.read(4 * 3))
struct.error: unpack requires a buffer of 12 bytes

Thank you!

struct.error: unpack requires a buffer of 12 bytes

There is a error about struct.error. Would you like tell me how to solve this error?thank you

bad performance on own data

Hi, i have tried to training your model on own dataset (59 images in total, outdoor scenarios).
The trained depth priors seem proper:


However, the NeRF cannot be trained at all. Its loss is not decreased and the PSNR is nearly unchanged. And finally, the trained NeRF just renders totally black images for both depth and rgb:

The experimental setups are the same as the ones in the default configuration.
Could you give some advices on this problem? Thanks at advance :)

Sharing COLMAP poses and depth maps

Hi, thank you very much for open-sourcing this project!

Would it be possible to share the ScanNet splits used in the paper (training and test), as well as the calculated COLMAP poses and sparse depth maps? I know we can calculate those ourselves, but it would go a long way in enabling a proper comparison with your method.

Again, thank you very much!

Training with own data

Hi,

I am trying to train NerfingMVS with my own data but I don't understand what is the "depth mask" that is used (folder: raw_data/scene*/depth/). How do I obtain this mask for my own data and is it also essential?

This is not the GT depth map used to train monocular depth network, right? The depth map obtained from MVS of colmap is used as GT, atleast thats what I understood from reading the paper.

Visualization of 3D voxel map

Hi, thanks for your remarkable work! After training a model, I want to visualize the voxel in the 3D space. Could you please give me some advice or related visualization codes? Many thanks.

The sampling in the code is different from that in the paper

In the paper: tn = D(1 − max(e, αl)), tf = D(1 + min(e, αh))
In the code:

near = (depth_priors * (1 - torch.clamp(depth_confidences, min=near_bound, max=far_bound))).unsqueeze(1)
far = (depth_priors * (1 + torch.clamp(depth_confidences, min=near_bound, max=far_bound))).unsqueeze(1))

Why do you clamp the confidence in the code?

Why sparse depth map resolution 288x364 ?

Hi,

I try to generate a larger resolution (eg. 480x640) depth prior map, however ,the visible results seems get worse the 288x364 resolution. Have you tried larger reslution depth prior or that's why you use sparse resolution depth prior.

Another question in your paper, I see the comparison results on the novel views , I'm wondering what's the exact novel view ,does it mean the the testing images or the rerendered views(eg. 120 poses) .Because I don't find the code of rendering the 120 poses images in your repo , which needs depth_prior and depth_confidence.

About the setting of depth_W and depth_H

Hi, I find that there is default setting of depth_W and depth_H in options.py. I would like to ask that if I use NerfingMVS on my custom dataset which has a totally different resolution from ScanNet, do I need to change their values or leave them not changed?

Thanks!

Could not fuse any points. FileNotFoundError dense.stereo/depth_maps/xx.jpg.geometric.bin

hi author! I have encountered problem in fusing points. And it failed in FileNotFoundError : [Errno 2] No such file or directory : './data/scene0004/dense.stereo/depth_maps/xx.jpg.geometric.bin' Could you please help me. Thank you so much!

Remove TorchsearchSorted module

Hi,
Thanks for your great work! I think you can replace the 'torchsearchsorted' module with 'torch.searchsorted()'. When I use NVIDIA GeForce RTX 2080 (same with you) to intall 'torchsearchsorted', I met some 'unsupported gpu architecture'. Maybe you can share how to install 'torchsearchsorted' on 2080 gpu with me.
This is a nice work. Thanks!

Mesh construction

Thanks for this excellent work!
I'm wondering whether we can generate the mesh model from depth map we get, and get the scene-level 3D reconstruction results?

关于深度和mask的一个问题

在depths, masks = load_colmap(image_list, args.datadir,args.depth_H, args.depth_W)这个函数里

def load_point_vis(path, masks):
with open(path, 'rb') as f:
n = struct.unpack('<Q', f.read(8))[0]
print('point number: {}'.format(n))
for i in range(n):
m = struct.unpack('<I', f.read(4))[0]
for j in range(m):
idx, u, v = struct.unpack('<III', f.read(4 * 3))
masks[idx][v, u] = 1

由于在linux装colmap没有装成功，所以在windows用colmap生成之后然后移过去，就没有用你们的脚本跑
但是在跑的时候在这一段出错了

请问您有遇到过吗 不知道是生成的数据的问题（没有在linux上用你们的脚本跑 但是我在windows下跑的时候也按照你们给定的参数设置了）

how to develop with colmap

Hi sorry to bother you
i want to integrate other keypoint detector and extractor in another matching method, but i dont have any experience with development in ubuntu, could you give me some suggestions with me
many thanks!!!

IndexError: list index out of range

when I follow the instructions and do the "sh run.sh scene0000_01" ,it was successful at first but when it comes to the :"Depths prior training begins !",it happens to occur the error as follow:
Depths prior training begins !
Dowloading https://storage.googleapis.com/mannequinchallenge-data/checkpoints/best_depth_Ours_Bilinear_inc_3_net_G.pth to /home/ycj/NerfingMVS/models/depth_priors/mc.pth ...
100% [....................................................] 21619796 / 21619796point number: 4696662
Traceback (most recent call last):
File "run.py", line 19, in
depth_priors.train(args)
File "/home/ycj/NerfingMVS/src/depth_priors.py", line 70, in train
args.depth_H, args.depth_W)
File "/home/ycj/NerfingMVS/utils/io_utils.py", line 27, in load_colmap
ply_masks = read_ply_mask(ply_path)
File "/home/ycj/NerfingMVS/utils/colmap_utils.py", line 49, in read_ply_mask
load_point_vis(ply_vis_path, masks)
File "/home/ycj/NerfingMVS/utils/colmap_utils.py", line 35, in load_point_vis
masks[idx][v, u] = 1
IndexError: list index out of range

could you tell me the reason?thx

what is the meaning of the parameters in configs?

sorry to bother.but I wanted to try to train my own data.I download the data from the web:https://vision.middlebury.edu/mview/data/
and I found you mentioned in "Run on Your Own Data!" says factor parameter controls the resolution of output depth maps.for example ,the resolution of the data I download is 640*480,so what number should the "factor" be in my configs?by the way ,could you tell me what does the other parameters as follows exactly mean?
N_rand = 1024
N_samples = 64
use_viewdirs = True
raw_noise_std = 1e0
thank you very much~

Question regarding guided sampling

Hi, thanks for open-sourcing the project

I have a question regarding these two lines:

NerfingMVS/src/run_nerf.py

Line 373 in 5890f15

near = (depth_priors * (1 - torch.clamp(depth_confidences, min=near_bound, max=far_bound))).unsqueeze(1)

NerfingMVS/src/run_nerf.py

Line 374 in 5890f15

far = (depth_priors * (1 + torch.clamp(depth_confidences, min=near_bound, max=far_bound))).unsqueeze(1)

I think you did not use the near_bound and far_bound as in equation (6) in the paper, is it correct, or I am missing something?

Issue about training

When the training finished, I face the problem

the code stopped.

CVD camparsion details

Hello :) I'm conducting experiments on ScanNet.
And the output results on final iteration are like this.

Can u explain the CVD experiments parameter**('frame_range')**?
I wanna to get a result you illustrated on your paper ...

Thanks

The question of how to install colmap

I'm sorry to bother you sir.I'm a student interested in your code,but I just don't know how to install it.
Before installing it into colmap folder,should I add

into it first?And then just follow the basic step to install the colmap into it without covering these files?I just don't understand what "do not cover colmap folder with the original version" and "we add masks to COLMAP and denote it as a submodule"means.I'm so looking forward to your reply.

how to use origin pose of scannet?

thanks for your great work!, I'm trying to use the origin pose files of scannet to modify NeRFingMVS, but I encountered a pose align problem: my result is always blurred, it seems as if the pose and the radiance field is a mismatch. I wonder if you are able to solve the problem. Thanks!

Link to results and checkpoints

Hi, I cannot find the link to the results and checkpoints of each scene, as advertised in the Usage section. Do you plan to release those as well?

Rendering only case - do we need depth priors?

Hello:) Thnx for sharing the code. Could you pls clarify: if we forward pass already learned scene to render views from poses of some random trajectory (e.g. circle), do we need depth priors?

Asking this because of this line from render_path() function in src/run_nerf.py:
rgb, disp, acc, depth, _ = render(H, W, focal, depth_priors=depth_priors[i], depth_confidences=depth_confidences[i], chunk=chunk, c2w=c2w[:3,:4], **render_kwargs)

And for poses with unseen view you utilize depth prior from the nearest seen view, correct?

运行自己的数据集时错误

log:
Post-colmap
Cameras 5
Images # 46
COLMAP fails for some images!

请问这个是什么问题呀

subprocess.CalledProcessError: Command 'mogrify -resize 50.0% -format png *.jpg' returned non-zero exit status 127.

After the process of depth prior training was done, a bug "/bin/sh: 1: mogrify: not found" occurred in "mogrify -resize 50.0% -format png *.jpg". How can I deal with it? Thank you!

Details for Novel View Synthesis

Thanks for your great work! I want to ask about some implementation details of the novel view synthesis.
In your paper, you mentioned, "While the proposed guided optimization strategy needs the adapted depth priors as input to guide point sampling along the camera ray, we can still perform novel view synthesis by directly using the adapted depth priors from the nearest seen view.", but I can't find the corresponding code in your repo and wonder how to use depth priors from the nearest seen view.
Look forward to your reply.

WARNING: Could not fuse any points.

When I run on my own data,it occured that error.However,it was alright on your datasets.


I trained it use four gpus,so I guessed that is not the problem.and I altered the depth_H and W to my picture format.
By the way.Before the fusion step,It occured configuration problem and PatchMatch::Problem.I just dont know what happened.

The size of tensor a(162) must match the size of tensor b(3) at non-singleton dimension 1

Hello, author! I have a question to ask you. When I run the code, I report the following errors. What is the reason? I didn't change the code.

How to get the 3D model

Hello， nice work， if I would like to get the model after nerf filter. what should i do.