kai-46 / arf-svox2 Goto Github PK

Hi, Kai-46, thanks for the impressive work! The proposed Deferred back-propagation method can render a full-resolution image in one batch. However, the back-propagation of the released code is implemented by the CUDA. Is it possible to accomplish it by Pytorch?

Hi sorry to bother you,
PS C:\Users\c8178\Github\ARF-svox2\opt> nvcc -V
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2023 NVIDIA Corporation
Built on Fri_Jan__6_19:04:39_Pacific_Standard_Time_2023
Cuda compilation tools, release 12.0, V12.0.140
Build cuda_12.0.r12.0/compiler.32267302_0

It seems I have installed nvcc already, but the svox2 error still exists.

Hi,
I saw on github that you doublet the code of the original author of plenoxels. And see you want to optimize depth. This is relevant to my research.In svox2.py file,define a volume_render_depth function?
I think this code is important:
*out_depth += (weight * ((t / opt.step_size) * ray.world_step);How to change this code to render zdepth?
But the original author said this is the absolute length along the ray, not the z-depth as usually expected.So how should I modify this code to get zdepth!

Hi, thanks for your great work!
I'm recently working on a topic concerning about the depth in scence, and I'm confused about the scale in the tnt dataset: Playground.

I could see that tmin is 0 and tmax is approximately 500 with a grid size of 640, while the scene is not that big as far as I can see in the point cloud.
I guess it's because of the world scale, but I am not quite sure. Could you give any comments on this?

Thanks a lot!

Hi, I have encontered CUDA out of memory after a few successful epochs, could you tell me how to solve this?

INFO:
my command line: bash try_tnt.sh Playground 1
my GPU: a Nvidia Titan RTX with 24G Memory

Bug snapshot:

Thank you in advance for you help!

在最后运行时，报的错，错误消息表明 load() 函数不支持 reset_basis_dim 这个关键字参数。这可能是因为您在调用 load() 函数时传递了一个不被支持的参数

想请问一下：如何能将这种大型的场景用原始nerf重建清晰，您的nerf++可以解决，但如果我想用原始nerf的话该如何解决这个问题，是调小相机轨迹半径还是什么操作？烦请您给点tips！谢谢

Hi! I was very intrigued by your work and started looking at your code.
(I'm not very good at cuda language)
I saw that you were measuring patch wise loss.
However, I'm having a hard time finding where exactly in your code, it calculates MSE loss between GT patch and output patch.

Thank you for your time.

Hi, thanks for your great work!
I'm recently working on a topic concerning about the depth in scence, and I'm confused about the scale in the tnt dataset.

I could see that tmin is 0 and tmax is approximately 500 with a grid size of 640, while the scene is not that big as far as I can see in the point cloud.
I guess it's because of the world scale, but I am not quite sure. Could you give any comments on this?

Thanks a lot!

Hi Kai, I hope you're doing well -- this is JJ! Congratulations on this great work!

I have a question regarding some parts of the code.
https://github.com/Kai-46/ARF-svox2/blob/22baf6e000e519b21ca26308f8ead1f4d2675cf5/opt/opt_style.py#L618
In this line, I see that you are downsampling the rendered image to half to compute the losses.
But you could instead render an image with a lower resolution, which would be much more memory efficient.
Is there a reason for doing this downsampling? Did you want to do anti-aliasing, perhaps?

Thanks in advance!

I encounter the following warning while running, which lead to subsequent Errors. How can I solve these？
I cancelled import svox in svox2.py, but still reported this warning.

warning:
/opt/svox2/utils.py:39: UserWarning: CUDA extension svox2.csrc could not be loaded! Operations will be slow.
Please do not import svox in the svox2 source directory.
warn("CUDA extension svox2.csrc could not be loaded! "

subsequent error 1:
Selecting random rays
Eval step
0%| | 0/3 [00:00<?, ?it/s]
Traceback (most recent call last):
File "opt.py", line 471, in
eval_step()
File "opt.py", line 406, in eval_step
rgb_pred_test = grid.volume_render_image(cam, use_kernel=True)
File "/opt/svox2/svox2.py", line 1210, in volume_render_image
and imrend_fn_name in _C.dict
AttributeError: 'NoneType' object has no attribute '__dict__'

subsequent error 2:
Traceback (most recent call last):
File "opt_style.py", line 439, in
assert os.path.isfile(args.init_ckpt), "must specify a initial checkpoint"
AssertionError: must specify a initial checkpoint

subsequent error 3:
Traceback (most recent call last):
File "render_imgs.py", line 114, in
grid = svox2.SparseGrid.load(args.ckpt, device=device)
File "/opt/svox2/svox2.py", line 1645, in load
z = np.load(path)
File "/root/miniconda3/lib/python3.8/site-packages/numpy/lib/npyio.py", line 405, in load
fid = stack.enter_context(open(os_fspath(file), "rb"))
FileNotFoundError: [Errno 2] No such file or directory: 'ckpt_arf/llff/fern_14.jpg/ckpt.npz/ckpt.npz'

If you could provide some suggestions, I would greatly appreciate it!

As described in create_env.sh, we use conda to create envs,
When I ran pip install . --upgrade --use-feature=in-tree-build, it prints:

Usage:
pip install [options] [package-index-options] ...
pip install [options] -r [package-index-options] ...
pip install [options] [-e] ...
pip install [options] [-e] ...
pip install [options] <archive url/path> ...
option --use-feature: invalid choice: 'in-tree-build' (choose from '2020-resolver', 'fast-deps')

Then ran with pip install . --upgrade --use-feature=2020-resolver', it turns out:

WARNING: --use-feature=2020-resolver no longer has any effect, since it is now the default dependency resolver in pip. This will become an error in pip 21.0.
Preparing metadata (setup.py) ... done
Building wheels for collected packages: svox2
Building wheel for svox2 (setup.py) ... error
error: subprocess-exited-with-error

× python setup.py bdist_wheel did not run successfully.
│ exit code: 1
╰─> [18 lines of output]

...

note: This error originates from a subprocess, and is likely not a problem with pip.
ERROR: Failed building wheel for svox2
Running setup.py clean for svox2
Failed to build svox2
Installing collected packages: svox2
Running setup.py install for svox2 ... error
error: subprocess-exited-with-error

× Running setup.py install for svox2 did not run successfully.
│ exit code: 1
╰─> [31 lines of output]

......

note: This error originates from a subprocess, and is likely not a problem with pip.
error: legacy-install-failure

× Encountered error while trying to install package.
╰─> svox2

note: This is an issue with the package mentioned above, not pip.
hint: See above for output from the failure.

Many THX!

Hi, your work is really amazing!
I wonder how we can customize the camera trajectory for rendering?
And another question, is there anyway to truncate the bounding box for the rendering scene and set the trimmed out parts as transparent?

Hi Kai,
Thanks for your great work! I m quite confused that why you still use the MSE loss before the NNFM loss in the style transfer stage in opt_style.py.

Hi,
Curious about the when ARF-NeRF will be release? Looking forward to it.

Since svox2 is based on CUDA, I wonder if there are any other kinds of NeRF models available such as original NeRF or TenSoRF?

Hi, I wonder to know that nerf++ can be used on LLFF dataset?

Hi，
when I try to run try{}.sh, it shows no module name sovx2, and I try to run pip install -e . according to https://github.com/sxyu/svox2, but there is an error subprocess.CalledProcessError: Command '['ninja', '-v']' returned non-zero exit status 1. could you help me fix it ,Thanks!

Excellent work!

I have created the envs, when running bash try_llff.sh flower 14, the result reports an error:

Traceback (most recent call last):
File "render_imgs.py", line 95, in
lpips_vgg = lpips.LPIPS(net="vgg").eval().to(device)
File "/anaconda3/envs/arf-svox2/lib/python3.8/site-packages/torch/nn/modules/module.py", line 927, in to
return self._apply(convert)
File "/anaconda3/envs/arf-svox2/lib/python3.8/site-packages/torch/nn/modules/module.py", line 579, in _apply
module._apply(fn)
File "/anaconda3/envs/arf-svox2/lib/python3.8/site-packages/torch/nn/modules/module.py", line 625, in _apply
self._buffers[key] = fn(buf)
File "/anaconda3/envs/arf-svox2/lib/python3.8/site-packages/torch/nn/modules/module.py", line 925, in convert
return t.to(device, dtype if t.is_floating_point() or t.is_complex() else None, non_blocking)
File "~/anaconda3/envs/arf-svox2/lib/python3.8/site-packages/torch/cuda/init.py", line 217, in _lazy_init
torch._C._cuda_init()
RuntimeError: CUDA driver initialization failed, you might not have a CUDA gpu.

I reinstalled cudatoolkit, but did not work. Excuting nvcc -V,

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2019 NVIDIA Corporation
Built on Wed_Oct_23_19:24:38_PDT_2019
Cuda compilation tools, release 10.2, V10.2.89

don't know what the problem is, THX!

When I try

. ./try_llff.sh fortress 2

or other scene name, the program will stop with error

Traceback (most recent call last):
  File "opt_style.py", line 765, in <module>
    train_step(optim_type="image")
  File "opt_style.py", line 640, in train_step
    rgb_pred_grad, loss_dict = compute_image_loss()
  File "opt_style.py", line 621, in compute_image_loss
    style_img.permute(2, 0, 1).unsqueeze(0),
RuntimeError: number of dims don't match in permute

and

Traceback (most recent call last):
  File "render_imgs.py", line 115, in <module>
    grid = svox2.SparseGrid.load(args.ckpt, device=device)
  File "[something omitted]/anaconda3/envs/arf-svox2/lib/python3.8/site-packages/svox2/svox2.py", line 1645, in load
    z = np.load(path)
  File "[something omitted]/anaconda3/envs/arf-svox2/lib/python3.8/site-packages/numpy/lib/npyio.py", line 428, in load
    fid = open(os_fspath(file), "rb")
FileNotFoundError: [Errno 2] No such file or directory: 'ckpt_arf/llff/room_2/ckpt.npz/ckpt.npz'

But there's no error with style 14. I didn't try other styles, and I haven't figured out the reason.

I'm currently using python=3.8.8, pytorch=1.7.0, torchvision=0.8.1, cudatoolkit=10.1 (the other packages are the same as environment.yml). I don't know whether it is caused by package version difference.

Hi there, I have been trying to use your work on my own dataset, I managed to get the colmap results and finished the voxel optimization process. However, I come across "AttributeError: 'NSVFDataset' object has no attribute 'render_c2w'" Error when rendering results using render_imgs.py. It seemed that the code automatically identify my dataset as a 'NSVFDataset' instead of a 'LLFFDataset'. Could you tell me how to solve this bug?

My dataset looks like this after the dataset generation process provided in svox2.

ERROR: Command errored out with exit status 1:
command: 'G:\anaconda3\python.exe' -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'H:\nerf\ARF-svox2-master1\ARF-svox2-master\setup.py'"'"'; file='"'"'H:\nerf\ARF-svox2-master1\ARF-svox2-master\set
up.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(file) if os.path.exists(file) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(comp
ile(code, file, '"'"'exec'"'"'))' bdist_wheel -d 'C:\Users\dell\AppData\Local\Temp\pip-wheel-6rlrf4nb'
cwd: H:\nerf\ARF-svox2-master1\ARF-svox2-master
Complete output (32 lines):
H:\nerf\ARF-svox2-master1\ARF-svox2-master\setup.py:25: UserWarning: The environment variable CUB_HOME was not found.Installation will fail if your system CUDA toolkit version is less than 11.NVIDIA CUB can be downloaded from h ttps://github.com/NVIDIA/cub/releases. You can unpack it to a location of your choice and set the environment variable CUB_HOME to the folder containing the CMakeListst.txt file.
warnings.warn(
G:\anaconda3\lib\site-packages\setuptools\installer.py:27: SetuptoolsDeprecationWarning: setuptools.installer is deprecated. Requirements should be satisfied by a PEP 517 installer.
warnings.warn(
running bdist_wheel
G:\anaconda3\lib\site-packages\torch\utils\cpp_extension.py:476: UserWarning: Attempted to use ninja as the BuildExtension backend but we could not find ninja.. Falling back to using the slow distutils backend.
warnings.warn(msg.format('we could not find ninja.'))
running build
running build_py
package init file 'svox2\csrc_init_.py' not found (or not a regular file)
running build_ext
G:\anaconda3\lib\site-packages\torch\utils\cpp_extension.py:358: UserWarning: Error checking compiler version for cl: [WinError 2] 系统找不到指定的文件。
warnings.warn(f'Error checking compiler version for {compiler}: {error}')
building 'svox2.csrc' extension
"C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.6\bin\nvcc" -c svox2/csrc/loss_kernel.cu -o build\temp.win-amd64-3.9\Release\svox2/csrc/loss_kernel.obj -IH:\nerf\ARF-svox2-master1\ARF-svox2-master\svox2\csrc\include -IG:\a
naconda3\lib\site-packages\torch\include -IG:\anaconda3\lib\site-packages\torch\include\torch\csrc\api\include -IG:\anaconda3\lib\site-packages\torch\include\TH -IG:\anaconda3\lib\site-packages\torch\include\THC "-IC:\Program Files
NVIDIA GPU Computing Toolkit\CUDA\v11.6\include" -IG:\anaconda3\include -IG:\anaconda3\Include "-IC:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.29.30133\ATLMFC\include" "-IC:\Program Files (x86)\Micr
osoft Visual Studio\2019\Community\VC\Tools\MSVC\14.29.30133\include" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\ucrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\shared" "-IC:\Program Files (x
86)\Windows Kits\10\include\10.0.19041.0\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\winrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\cppwinrt" -Xcudafe --diag_suppress=dll_interface_confl
ict_dllexport_assumed -Xcudafe --diag_suppress=dll_interface_conflict_none_assumed -Xcudafe --diag_suppress=field_without_dll_interface -Xcudafe --diag_suppress=base_class_has_different_dll_interface -Xcompiler /EHsc -Xcompiler /wd4
190 -Xcompiler /wd4018 -Xcompiler /wd4275 -Xcompiler /wd4267 -Xcompiler /wd4244 -Xcompiler /wd4251 -Xcompiler /wd4819 -Xcompiler /MD -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__
CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=csrc -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 --use-local-env
G:/anaconda3/lib/site-packages/torch/include\c10/macros/Macros.h(138): warning C4067: 预处理器指令后有意外标记 - 应输入换行符
G:\anaconda3\lib\site-packages\torch\include\pybind11\cast.h(624): error: too few arguments for template template parameter "Tuple"
detected during instantiation of class "pybind11::detail::tuple_caster<Tuple, Ts...> [with Tuple=std::pair, Ts=<T1, T2>]"
(721): here

G:\anaconda3\lib\site-packages\torch\include\pybind11\cast.h(717): error: too few arguments for template template parameter "Tuple"
detected during instantiation of class "pybind11::detail::tuple_caster<Tuple, Ts...> [with Tuple=std::pair, Ts=<T1, T2>]"
(721): here

2 errors detected in the compilation of "svox2/csrc/loss_kernel.cu".
loss_kernel.cu
error: command 'C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.6\bin\nvcc.exe' failed with exit code 1
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
File "G:\anaconda3\lib\site-packages\colorama\ansitowin32.py", line 59, in closed
return stream.closed
ValueError: underlying buffer has been detached

ERROR: Failed building wheel for svox2
Running setup.py clean for svox2
Failed to build svox2
Installing collected packages: svox2
Running setup.py install for svox2 ... error
ERROR: Command errored out with exit status 1:
command: 'G:\anaconda3\python.exe' -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'H:\nerf\ARF-svox2-master1\ARF-svox2-master\setup.py'"'"'; file='"'"'H:\nerf\ARF-svox2-master1\ARF-svox2-master\s
etup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(file) if os.path.exists(file) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(co
mpile(code, file, '"'"'exec'"'"'))' install --record 'C:\Users\dell\AppData\Local\Temp\pip-record-q3vs7v9d\install-record.txt' --single-version-externally-managed --compile --install-headers 'G:\anaconda3\Include\svox2'
cwd: H:\nerf\ARF-svox2-master1\ARF-svox2-master
Complete output (47 lines):
H:\nerf\ARF-svox2-master1\ARF-svox2-master\setup.py:25: UserWarning: The environment variable CUB_HOME was not found.Installation will fail if your system CUDA toolkit version is less than 11.NVIDIA CUB can be downloaded from
https://github.com/NVIDIA/cub/releases. You can unpack it to a location of your choice and set the environment variable CUB_HOME to the folder containing the CMakeListst.txt file.
warnings.warn(
G:\anaconda3\lib\site-packages\setuptools\installer.py:27: SetuptoolsDeprecationWarning: setuptools.installer is deprecated. Requirements should be satisfied by a PEP 517 installer.
warnings.warn(
running install
G:\anaconda3\lib\site-packages\setuptools\command\install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
warnings.warn(
running build
running build_py
creating build
creating build\lib.win-amd64-3.9
creating build\lib.win-amd64-3.9\svox2
copying svox2\defs.py -> build\lib.win-amd64-3.9\svox2
copying svox2\setup.py -> build\lib.win-amd64-3.9\svox2
copying svox2\svox2.py -> build\lib.win-amd64-3.9\svox2
copying svox2\utils.py -> build\lib.win-amd64-3.9\svox2
copying svox2\version.py -> build\lib.win-amd64-3.9\svox2
copying svox2_init_.py -> build\lib.win-amd64-3.9\svox2
package init file 'svox2\csrc_init_.py' not found (or not a regular file)
running build_ext
G:\anaconda3\lib\site-packages\torch\utils\cpp_extension.py:476: UserWarning: Attempted to use ninja as the BuildExtension backend but we could not find ninja.. Falling back to using the slow distutils backend.
warnings.warn(msg.format('we could not find ninja.'))
G:\anaconda3\lib\site-packages\torch\utils\cpp_extension.py:358: UserWarning: Error checking compiler version for cl: [WinError 2] 系统找不到指定的文件。
warnings.warn(f'Error checking compiler version for {compiler}: {error}')
building 'svox2.csrc' extension
creating build\temp.win-amd64-3.9
creating build\temp.win-amd64-3.9\Release
creating build\temp.win-amd64-3.9\Release\svox2
creating build\temp.win-amd64-3.9\Release\svox2\csrc
"C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.6\bin\nvcc" -c svox2/csrc/loss_kernel.cu -o build\temp.win-amd64-3.9\Release\svox2/csrc/loss_kernel.obj -IH:\nerf\ARF-svox2-master1\ARF-svox2-master\svox2\csrc\include -IG:
\anaconda3\lib\site-packages\torch\include -IG:\anaconda3\lib\site-packages\torch\include\torch\csrc\api\include -IG:\anaconda3\lib\site-packages\torch\include\TH -IG:\anaconda3\lib\site-packages\torch\include\THC "-IC:\Program File
s\NVIDIA GPU Computing Toolkit\CUDA\v11.6\include" -IG:\anaconda3\include -IG:\anaconda3\Include "-IC:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.29.30133\ATLMFC\include" "-IC:\Program Files (x86)\Mi
crosoft Visual Studio\2019\Community\VC\Tools\MSVC\14.29.30133\include" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\ucrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\shared" "-IC:\Program Files
G:\anaconda3\lib\site-packages\torch\include\pybind11\cast.h(717): error: too few arguments for template template parameter "Tuple"
detected during instantiation of class "pybind11::detail::tuple_caster<Tuple, Ts...> [with Tuple=std::pair, Ts=<T1, T2>]"
(721): here

2 errors detected in the compilation of "svox2/csrc/loss_kernel.cu".
loss_kernel.cu
error: command 'C:\\Program Files\\NVIDIA GPU Computing Toolkit\\CUDA\\v11.6\\bin\\nvcc.exe' failed with exit code 1
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "G:\anaconda3\lib\site-packages\colorama\ansitowin32.py", line 59, in closed
    return stream.closed
ValueError: underlying buffer has been detached
----------------------------------------

ERROR: Command errored out with exit status 1: 'G:\anaconda3\python.exe' -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'H:\nerf\ARF-svox2-master1\ARF-svox2-master\setup.py'"'"'; file='"'"'H:\nerf\ARF-s
vox2-master1\ARF-svox2-master\setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(file) if os.path.exists(file) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"'
, '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' install --record 'C:\Users\dell\AppData\Local\Temp\pip-record-q3vs7v9d\install-record.txt' --single-version-externally-managed --compile --install-headers 'G:
anaconda3\Include\svox2' Check the logs for full command output.

I wanted to give it randomness, so I gave the standard deviation differently, but it doesn't seem to apply to the results. What should I do?

I try to follow the guide and run the optimized code (opt_style.py).
I think I shall have an init checkpoint first, I want to know where can I download it?

line 439:
assert os.path.isfile(args.init_ckpt), "must specify a initial checkpoint"

Sorry if there is anything I missed.