sjtuytc / unboundednerfpytorch Goto Github PK

你知道怎么解决么，感谢万分

Hi,
I found the depth value is a normalize between 0 and 1 when I debug the inference code.
How to get the real world scale depth?

I think this code is used to visualize mesh, but how should this code be used? How to generate mesh data?

eval_block_nerf.py里面，depth_video_writer.write(Depth_compose['compose'])，直接将1通道的深度图写入视频，我发现结果视频都没法播放，好像是因为cv2.VideoWriter没法直接将1通道的图片合成视频，需要先转为3通道，所以我在前面加了Depth_compose['compose'] = cv2.cvtColor(Depth_compose['compose'],cv2.COLOR_GRAY2BGR)，就可以成功转成视频了。

Can not find pretrained model weights via https://drive.google.com/drive/folders/1Lcc6MF35EnXGyUy0UZPkUx7SfeLsv8u9 in README file? Could you please tell me where to find them?

Hi,

I am wondering that what is the difference between Block-nerf and Mega-nerf?

Thanks and regards,

Hung

您好，我在自制数据集运行时出现了一些问题，想跟您请教一下。

背景：
我是在实验室内拍摄了实验室一周（包括两排桌椅、电脑以及天花板、地板）照片，有水平移动拍摄的、有俯拍和仰拍的，也有环绕着拍摄的，一共411张图。
对于拍摄的图片，利用colmap保存了database.db，然后进行了特征提取、特征匹配以及稀疏重建，最后Export model将其保存在"sparse/0/"文件夹下。
利用Fyusion/LLFF将导出的位姿数据转换成llff数据格式，由于照片数（411）和实际在colmap注册的数量（401）不匹配，因此出现了ERROR: the correct camera poses for current points cannot be accessed 这个错误，利用Fyusion/LLFF#60 (comment) 解决了问题，得到了poses_bounds.npy，并删掉了colmap未注册的照片。

问题：
运行 python run.py --config configs/custom/room.py 时，报错如下：

Loading images from data/room/dense/images_2
Loaded image data (2880, 1620, 3, 401) [2880. 1620. 2119.14228993]
Loaded data/room/dense 8.725662744276601e-10 38.88514969557468
recentered (3, 5)
[[ 1.0000000e+00 7.9511068e-08 7.2999038e-09 -8.4269325e+01]
[-7.9511068e-08 1.0000000e+00 -1.7426238e-07 3.5367581e+02]
[-7.2999176e-09 1.7426238e-07 1.0000000e+00 -1.4172569e+02]]
/home/vcis6/Userlist/Zouchen/LargeScaleNeRFPytorch/lib/load_llff.py:409: UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor. (Triggered internally at /opt/conda/conda-bld/pytorch_1634272204863/work/torch/csrc/utils/tensor_new.cpp:201.)
render_poses = torch.Tensor(render_poses)
Data:
(401, 3, 5) (401, 2880, 1620, 3) (401, 2)
HOLDOUT view is 214
Loaded llff (401, 2880, 1620, 3) torch.Size([120, 3, 5]) [2880. 1620. 2119.1423] data/room/dense
DEFINING BOUNDS
NEAR FAR 0.0 1.0
train: start
compute_bbox_by_cam_frustrm: start
/home/vcis6/anaconda3/envs/mega-nerf/lib/python3.9/site-packages/torch/functional.py:445: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at /opt/conda/conda-bld/pytorch_1634272204863/work/aten/src/ATen/native/TensorShape.cpp:2157.)
return _VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined]
compute_bbox_by_cam_frustrm: xyz_min tensor([nan, nan, nan])
compute_bbox_by_cam_frustrm: xyz_max tensor([nan, nan, nan])
compute_bbox_by_cam_frustrm: finish
train: skip coarse geometry searching
scene_rep_reconstruction (fine): train from scratch
scene_rep_reconstruction (fine): use multiplane images
dmpigo: world_size tensor([-9223372036854775808, -9223372036854775808, 256])
dmpigo: voxel_size_ratio 1.0
Traceback (most recent call last):
File "/home/vcis6/Userlist/Zouchen/LargeScaleNeRFPytorch/run.py", line 630, in
train(args, cfg, data_dict)
File "/home/vcis6/Userlist/Zouchen/LargeScaleNeRFPytorch/run.py", line 562, in train
scene_rep_reconstruction(
File "/home/vcis6/Userlist/Zouchen/LargeScaleNeRFPytorch/run.py", line 328, in scene_rep_reconstruction
model, optimizer = create_new_model(cfg, cfg_model, cfg_train, xyz_min, xyz_max, stage, coarse_ckpt_path)
File "/home/vcis6/Userlist/Zouchen/LargeScaleNeRFPytorch/run.py", line 266, in create_new_model
model = dmpigo.DirectMPIGO(
File "/home/vcis6/Userlist/Zouchen/LargeScaleNeRFPytorch/lib/dmpigo.py", line 41, in init
self.density = grid.create_grid(
File "/home/vcis6/Userlist/Zouchen/LargeScaleNeRFPytorch/lib/grid.py", line 29, in create_grid
return DenseGrid(**kwargs)
File "/home/vcis6/Userlist/Zouchen/LargeScaleNeRFPytorch/lib/grid.py", line 45, in init
self.grid = nn.Parameter(torch.zeros([1, channels, *world_size]))
RuntimeError: Trying to create tensor with negative dimension -9223372036854775808: [1, 1, -9223372036854775808, -9223372036854775808, 256]

（在这之前，我在单个方向拍了38张照片，然后利用上面相同的步骤，可以顺利生成模型以及视频）

hi,I reproduced the scene you provided,but the results are vague,what should i do ?

Hello,
Thanks for your effort for reproducing Block-Nerf.
I use the scripts in fetch_data_from_tf_record.py to generate dataset for Block-Nerf. I observed that you use the proposed intrinsics to generate rays in camera coordinate (Line 133) while the output intrinsics (fx, fy) are the provided intrinsics + [W/2, H/2] (Line 145). It is confusing and could you explain it for me?

Thanks again.

About NeRF generalization, as I have tested, if I use block_0 model to inference block_8 dataset, it's hard to get worthy result. So, the nerf model generalization is really like all kinds of references expressed that nerf's generalization is very poor? If we expand the block area, that means to train more data, e.g. treat one city dataset as one block, then the generalization is better or not?

Nvidia has published instant-ngp, a refinement of the NeRF algorithm that allows for training times 2-3 orders of magnitude faster than traditional NeRFs.

Could Block-NeRF be adapted to use the same optimizations? My iteration speed would be much faster if it didn't take hours-days every time I wanted to experiment with a new set of collections! 😃

GPU：A6000
block_nerf_eval.sh：python eval_block_nerf.py --chunk 8192 # 3090ti
eval_block_nerf.py：

ckpy_path：https://drive.google.com/drive/folders/1O7uzcPBQHNAcmAcmcS6TRbLqiIDE3D0y

问题：都是按照仓库下载下来的直接运行，但是跑出来的result_pytorch_waymo的depth_images都是下面这种

请问是哪里设置出问题了吗？期待您的答复！

5.Build your custom large-scale NeRF,is it Block-NeRF or Mega-NeRF or others?Look forward to your reply.

Hi, many thanks for your efforts in these works!
I run the training code on the waymo dataset following your instructions and modified some configuration to save the CPU and GPU memory (chunk size 100, batch size 512). However, a problem arises that the training metrics are not finite and a debugging process stops the training code.

In Pdb, the dictionary metric contains four key-value pair, i.e., psnr, depth_variance, photo_loss and loss, but their values are all nan. And the values item['rgbs'] and item['rays] seem proper.

Looking forward to your reply! Thank you at advance!

The existing NeRF papers illustrated that we can use COLMAP to generate transform.json file. However, it comes with a condition that there should be 70% overlap with each images.

In Block-NeRF, they mentioned that image data captured from 12 cameras that collectively provide a 360° view. 8 of the cameras provide a complete surround view from the roof of the car, with 4 additional cameras located at the vehicle front pointing forward and sideways.

Would it be possible to share the piece of code that used those 12 camera images as input to generate a transform.json ?

Many thanks.

I'm running into this problem please ask when the cuda version is not compatible with the problem

this is my configuration:
2080TI,cuda10.1,pytorch==1.7.0 torchvision==0.8.0 torchaudio==0.7.0 cudatoolkit=10.1 ( torch==1.7.1+cu101 torchvision==0.8.2+cu101 torchaudio==0.7.2)

Hello! I am also recently trying to replicate Block-NeRF and have built the most basic network architecture, including merging Mip-NeRF with NeRF-Wild, and got good results, here is my email: [email protected], if interested you can email me and I will my current progress

hi,Thanks for your excellent work, but I get stuck in some places when I train the module, when the video memory is occupied, but no gpu is called to participate in the calculation, I don't know how to deal with this situation, I hope you can give some advice

.

Hi, Thanks for your great works!
I have downloaded your preprocessed waymo dataset pytorch_block_nerf_data.zip and run the code to generate the masks for training following your instructions. However, the code throw an error about missing corrdinate.pt. Could you give some advice about this problem? Is the official waymo dataset necessary for training?
Looking forward to your reply.
Thanks.

作者，你好
我用自采数据生成Block_nerf的训练数据，遇到下面问题。用github上面的sample data，就不会出现。
采用这个脚本 python tools/imgs2poses.py data/front_left

Undistorting image [1/2]
Undistorting image [2/2]
Writing reconstruction...
Writing configuration...
Writing scripts...
Elapsed time: 0.004 [minutes]
Undistort images
Finished running COLMAP! Congrats!
Post-colmap
Cameras 5
Images # 2
ERROR: the correct camera poses for current points cannot be accessed
Factors: [2, 4, 8]
Minifying 2 data/front_left/dense
mogrify -resize 50% -format png *.png
Done
Minifying 4 data/front_left/dense
mogrify -resize 25% -format png *.png
Done
Minifying 8 data/front_left/dense
mogrify -resize 12% -format png *.png
Done
Done with imgs2poses

Hi,
Why use the transpose matrix of the cam_to_world rotate matrix in the line?
I remember the transpose matrix of a rotate matrix is equal to the inverse matrix. The inverse matrix is the mean from the world coordinate to the camera coordinate.

hi, thanks for this great work for large scene.
I'm a Chinese college student, when I want to reproduce your block-nerf, run "bash scripts/block_nerf_eval.sh" and say "No module named block_nerf.rendering", there is really no rendering.py under block-nerf, didn't you put it up?

when i generate a fly-through video it make success, but when i download it and open cause file damage? I dont know what's wrong, could you help me?

Thanks for your great work! Could you please provide google drive links for waymo dataset, because I cannot download it with 'gdown --id". Thanks in advance!

Hi, do u have any engineering experience for improving the depth training.if the pose is correct, do I need tune the params?

Here i got in own dataset by default setting

Here from pretrained model in supported dataset

Hi, First of all thank you for your great work.
When I was about to train the block-nerf model on my own data set, I followed your advice and validated it on the data set you provided and the pre-training model, but when I ran: bash scripts/block_nerf_eval.sh. The following error occurs:

I check the block_nerf file in the project, I do not find related function, including block_nerf_model、block_nerf_lightning. I guess there is something wrong with the environment. Can you help me? Thanks.

问题1：ModuleNotFoundError: No module named 'block_nerf.block_nerf_lightning'
问题2：将block_nerf.block_nerf_lightning用到的地方注释后会提示
block_nerf/block_nerf_model.py", line 20, in init
self.loss = BlockNeRFLoss(1e-2) # hparams['Visi_loss']
NameError: name 'BlockNeRFLoss' is not defined

请问：block_nerf是还缺少依赖文件吗，感谢您的后续解答

Hello, thank you for your great works! I want to try my own data on block-nerf, so I want to create a dataset similar to Waymo Block-NeRF Dataset. May I know that how is ray_origins and ray_directions generated? Is there any method we can get r_o and r_d, if we have pose and camera intrinsics from SLAM?

When I run this code on Madoka, I meet this problem. The bin file doesn't generate, So the code raise the error FileNotFoundError.

https://github.com/sjtuytc/LargeScaleNeRFPytorch/blob/d6e7371d4c11227711e021a30ff43f300fdcf8bb/tools/colmap_utils/pose_utils.py#L15

之前我们讨论是借鉴nerf--来做位姿优化。但我有两个问题：
1.如何处理大场景的位姿优化，因为汽车行驶过程中，可能后期见到的画面和之前的图像没有任何形似点。
2.怎么更好地利用视频序列来做位姿优化。
谢谢您的工作！

I really appreciate for making this repo public!

I was wondering how to get a "sequential" list of trainings per camera (out of 12 cameras).

Here is how I understood the data framework and tried to get a sequential list of images:

• the data has been preprocessed by globing the original data by its filename (which I assumed to include sequential information).
• The preprocessing script recorded this globbed sequence using 'index' key in metadata dictionary.
• Meanwhile, each image is captured from one of 12 cameras, whose index is stored as 'cam_idx' in metadata dictionary.
• So, I sub-sampled images with same 'cam_idx' per block, and ordered them by 'index'.

This gave me pretty random sequence of images... any hints on retrieving clean sequence as in "oct2_124_300_frames_trim.mp4" video that shows training frames sequentially in the README.md?

Thank you in advance!

Hello, many thanks for your great work!
I found that the pytorch_block_nerf_dataset is not in a format similar to mega-nerf. I don't know whether this dataset could be directly used for training by your codes. Could you provide several hands-on commands for that? If it could not be directly trained, could you give some hints on how to convert it to mega-nerf format? Thank you very much!

hi,i have a questions ,how to implement the parallel training and merging function in blocknerf when it needs to run large dataset?

I downLoad the waymodataset from https://drive.google.com/drive/folders/1Lcc6MF35EnXGyUy0UZPkUx7SfeLsv8u9, run
export BLOCK_INDEX=0
bash scripts/block_nerf_train.sh ${BLOCK_INDEX} ,but has a promblem.

非常感谢大佬的无私分享。想训练自己的数据，但不知到config要怎么写，下面是configs/custom/Otobai.py的一部分，大佬能给点指导吗？感谢，感谢
factor=2,
movie_render_kwargs={
'scale_r': 0.8,
'scale_f': 10.0,
'zrate': 6.0,
'zdelta': 0.5,
}

4.3 Train sub-modules.这一步的输出是block-nerf的网络模型权重参数？，没有三维场景模型或者视频输出吗？后面也没看到街景建模的三维场景模型，最后的生成只是三维场景的视频？

Hi, I have an existing city-scale (1km by 1km) colmap model of 20'000 grayscale images with well refined poses. I am a bit lost on how to train using this data.

How do I convert this into the format required for training? I have cameras.bin, images.bin, points3D.bin and the corresponding images. Additionally, what config file do you recommend for such a scenario? Finally, I suspect not, but does this project support sparse depth supervision?

I think the major part of this project is only a copy of Mega-NeRF. While in the code and README, this repo uses BlockNeRF and claims it supports BlockNeRF, which is actually not true and can be misleading to others. I think this issue should be fixed.

你好，你计划什么时候完成blcok-nerf的全部工作，谢谢

Run with their own data, the effect is poor, can't see 3 d effect, very fuzzy。

https://user-images.githubusercontent.com/13003282/203454463-57098029-f8a0-43db-80e1-12449d1dcab4.mp4

After visualize the waymo's training pose, here we can see, y axis is the car driving direction. If in my dataset z axis is the car driving direction, do I need transform the pose same as waymo training pose?

Hello ,I'm a student from HUST . Recently I'm running Mega-NeRF on Waymo Dataset and trying to do some experiments. Knowing that this project have finished the task ,so I try to running the program on this code hub. Unfortunately ,the script callback a error explained that No such file or directory.
I think I follow the correct steps ,firstly I download the Waymo dataset ,and run the script get_one_block_meta_and_visualize.py and split_block.py.I found the file coordinate.pt do not exists exactly .I'm confused , I don't know what step is wrong. Can you help me?

Traceback (most recent call last):
  File "create_cluster_masks.py", line 214, in <module>
    main(_get_mask_opts())
  File "/environment/miniconda3/lib/python3.7/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 345, in wrapper
    return f(*args, **kwargs)
  File "/environment/miniconda3/lib/python3.7/site-packages/torch/autograd/grad_mode.py", line 28, in decorate_context
    return func(*args, **kwargs)
  File "create_cluster_masks.py", line 57, in main
    coordinate_info = torch.load(dataset_path / 'coordinates.pt', map_location='cpu')
  File "/environment/miniconda3/lib/python3.7/site-packages/torch/serialization.py", line 594, in load
    with _open_file_like(f, 'rb') as opened_file:
  File "/environment/miniconda3/lib/python3.7/site-packages/torch/serialization.py", line 230, in _open_file_like
    return _open_file(name_or_buffer, mode)
  File "/environment/miniconda3/lib/python3.7/site-packages/torch/serialization.py", line 211, in __init__
    super(_open_file, self).__init__(open(name, mode))
FileNotFoundError: [Errno 2] No such file or directory: '~/data/data/pytorch_waymo_dataset/coordinates.pt'

I have tried colmap, but I meet the error "ERROR: the correct camera poses for current points cannot be accessed".

Thanks for your time.

Hiii I met a weird problem when following your recommended commands. Since I have already downloaded the corresponding dataset from the provided google drive link in this repo, the file structure specified in the load_waymo.py might be different.

When I ran "python run_FourierGrid.py --program train --config FourierGrid/configs/waymo/waymo_no_block.py --num_per_block 100 --render_video --exp_id 30", I got "No such file or directory: 'data/sep19_ordered_dataset/metadata.json'".

However, I don't think a folder called "sep19_ordered_dataset" exists in pytorch_waymo_dataset. Maybe I miss something and I will appreciate it if you could give some advice.
Thanks:)

Thanks for great work! I once used waymo open dataset for blocknerf to reconstruct city scene. However it shows that there is no near and far plane information in its dataset, which seems necessary in training nerf model. I wonder how to deal with it.

你好，waymo数据集上的训练大该什么时候会修复呢

问题提示：
block-nerf/block_nerf/rendering.py", line 19, in get_cone_mean_conv
* mean_t ** 2 - difference_t ** 2) / (3 * mean_t ** 2
UnboundLocalError: local variable 'mean_t' referenced before assignment

关于get_cone_mean_conv函数那里确实没有定义mean_t

感谢您的后续解答

I'm trying to generate a Block-NeRF with my own data

How would you proceed if you were to split your own data into separate blocks, in order to train them with train_block_nerf.py? split_block.py can't run on custom data because of lacking files such as train_all_meta.pt and split_block_train.json which are included in Waymo's data. Is there a way to generate these required files?