How to optimize for rendering about adop

[DenShlk]

Hi! I guess you'll have to tweak render_scale parameter in adop_viewer. I didn't found the way to do it from cli, but you can change it's default value in adop_viewer.h file and recompile.

[Ilyaprok]

Hi! I guess you'll have to tweak render_scale parameter in adop_viewer. I didn't found the way to do it from cli, but you can change it's default value in adop_viewer.h file and recompile.

Hi! Thanks, I have tried this, but unsucceded.

terminate called after throwing an instance of 'c10::CUDAOutOfMemoryError' what(): CUDA out of memory. Tried to allocate 136.00 MiB (GPU 0; 3.82 GiB total capacity; 1.75 GiB already allocated; 25.06 MiB free; 1.92 GiB reserved in total by PyTorch) Exception raised from malloc at ../c10/cuda/CUDACachingAllocator.cpp:438 (most recent call first): frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x6c (0x7f2688abe7ac in /home/robot/anaconda3/envs/adop/lib/python3.9/site-packages/torch/lib/libc10.so) frame #1: <unknown function> + 0x1c5e4 (0x7f26886df5e4 in /home/robot/anaconda3/envs/adop/lib/python3.9/site-packages/torch/lib/libc10_cuda.so) frame #2: <unknown function> + 0x1dbec (0x7f26886e0bec in /home/robot/anaconda3/envs/adop/lib/python3.9/site-packages/torch/lib/libc10_cuda.so) frame #3: <unknown function> + 0x1e1e5 (0x7f26886e11e5 in /home/robot/anaconda3/envs/adop/lib/python3.9/site-packages/torch/lib/libc10_cuda.so) frame #4: at::native::empty_cuda(c10::ArrayRef<long>, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, c10::optional<c10::MemoryFormat>) + 0x103 (0x7f268bbd2da3 in /home/robot/anaconda3/envs/adop/lib/python3.9/site-packages/torch/lib/libtorch_cuda.so) frame #5: <unknown function> + 0x3248d05 (0x7f268bd41d05 in /home/robot/anaconda3/envs/adop/lib/python3.9/site-packages/torch/lib/libtorch_cuda.so) frame #6: at::meta::upsample_bilinear2d::meta(at::Tensor const&, c10::ArrayRef<long>, bool, c10::optional<double>, c10::optional<double>) + 0xf7 (0x7f26dd839857 in /home/robot/anaconda3/envs/adop/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so) frame #7: <unknown function> + 0x3250935 (0x7f268bd49935 in /home/robot/anaconda3/envs/adop/lib/python3.9/site-packages/torch/lib/libtorch_cuda.so) frame #8: <unknown function> + 0x3250a21 (0x7f268bd49a21 in /home/robot/anaconda3/envs/adop/lib/python3.9/site-packages/torch/lib/libtorch_cuda.so) frame #9: at::redispatch::upsample_bilinear2d(c10::DispatchKeySet, at::Tensor const&, c10::ArrayRef<long>, bool, c10::optional<double>, c10::optional<double>) + 0xf8 (0x7f26ddf955e8 in /home/robot/anaconda3/envs/adop/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so) frame #10: <unknown function> + 0x37f83ef (0x7f26dfbd93ef in /home/robot/anaconda3/envs/adop/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so) frame #11: <unknown function> + 0x37f8614 (0x7f26dfbd9614 in /home/robot/anaconda3/envs/adop/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so) frame #12: at::upsample_bilinear2d(at::Tensor const&, c10::ArrayRef<long>, bool, c10::optional<double>, c10::optional<double>) + 0x16d (0x7f26ddc677cd in /home/robot/anaconda3/envs/adop/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so) frame #13: <unknown function> + 0x43a76f7 (0x7f26e07886f7 in /home/robot/anaconda3/envs/adop/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so) frame #14: torch::nn::UpsampleImpl::forward(at::Tensor const&) + 0x52 (0x7f26e0784092 in /home/robot/anaconda3/envs/adop/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so) frame #15: torch::nn::AnyModuleHolder<torch::nn::UpsampleImpl, at::Tensor const&>::forward(std::vector<torch::nn::AnyValue, std::allocator<torch::nn::AnyValue> >&&) + 0x2d3 (0x7f26ed5bc793 in /home/robot/ADOP/build/bin/libNeuralPoints.so) frame #16: torch::nn::AnyValue torch::nn::AnyModule::any_forward<at::Tensor&>(at::Tensor&) + 0x9f (0x7f26ed598fdf in /home/robot/ADOP/build/bin/libNeuralPoints.so) frame #17: at::Tensor torch::nn::SequentialImpl::forward<at::Tensor, at::Tensor&>(at::Tensor&) + 0x4e (0x7f26ed5990ee in /home/robot/ADOP/build/bin/libNeuralPoints.so) frame #18: Saiga::UpsampleBlockImpl::forward(std::pair<at::Tensor, at::Tensor>, std::pair<at::Tensor, at::Tensor>) + 0x92 (0x7f26ed59ea12 in /home/robot/ADOP/build/bin/libNeuralPoints.so) frame #19: Saiga::MultiScaleUnet2dImpl::forward(Saiga::ArrayView<at::Tensor>, Saiga::ArrayView<at::Tensor>) + 0x981 (0x7f26ed59f901 in /home/robot/ADOP/build/bin/libNeuralPoints.so) frame #20: NeuralPipeline::Forward(NeuralScene&, std::vector<std::shared_ptr<TorchFrameData>, std::allocator<std::shared_ptr<TorchFrameData> > >&, at::Tensor, bool, bool, float, Eigen::Matrix<float, 3, 1, 0, 3, 1>) + 0xe21 (0x7f26ed5895b1 in /home/robot/ADOP/build/bin/libNeuralPoints.so) frame #21: RealTimeRenderer::Render(ImageInfo) + 0xb73 (0x7f26ed5c7ac3 in /home/robot/ADOP/build/bin/libNeuralPoints.so) frame #22: <unknown function> + 0x1d837 (0x55bfcff6c837 in ./build/bin/adop_viewer) frame #23: Saiga::DeferredRenderer::renderGL(Saiga::Framebuffer*, Saiga::ViewPort, Saiga::Camera*) + 0xaf5 (0x7f26ed24e925 in /home/robot/ADOP/build/External/saiga/lib/libsaiga_opengl.so) frame #24: Saiga::OpenGLRenderer::render(Saiga::RenderInfo const&) + 0x150 (0x7f26ed2972b0 in /home/robot/ADOP/build/External/saiga/lib/libsaiga_opengl.so) frame #25: Saiga::WindowBase::render() + 0x4c (0x7f26ec61a67c in /home/robot/ADOP/build/External/saiga/lib/libsaiga_core.so) frame #26: Saiga::MainLoop::render(float, float) + 0x7a (0x7f26ec618d4a in /home/robot/ADOP/build/External/saiga/lib/libsaiga_core.so) frame #27: Saiga::MainLoop::startMainLoop(Saiga::MainLoopParameters) + 0x25c (0x7f26ec61912c in /home/robot/ADOP/build/External/saiga/lib/libsaiga_core.so) frame #28: Saiga::WindowBase::startMainLoop(Saiga::MainLoopParameters) + 0x2b (0x7f26ec61aaab in /home/robot/ADOP/build/External/saiga/lib/libsaiga_core.so) frame #29: <unknown function> + 0x1945c (0x55bfcff6845c in ./build/bin/adop_viewer) frame #30: __libc_start_main + 0xf3 (0x7f26887170b3 in /lib/x86_64-linux-gnu/libc.so.6) frame #31: <unknown function> + 0x1986e (0x55bfcff6886e in ./build/bin/adop_viewer) Aborted (core dumped)

Ubuntu 20.04 (not WSL)
GeForce 1650

[DenShlk]

What scene and values you've tried?
P.S. I'd use ``` to format your error message, otherwise github parses #<number> as references to issues.

[Ilyaprok]

What scene and values you've tried?

I tried:
render_scale =0.5,
render_scale =0.2,
render_scale =0.05,
render_scale =0.01.
Scenes:
tt_lighthouse,
tt_m60,
tt_train.
for example render_scale = 0.5, scene=tt_train

[DenShlk]

Am I right that it crashes after showing some number of frames?
Check that there are no other apps which use significant amount of gpu mem. (nvidia-smi console command)

[Ilyaprok]

It crashes after switching on neuro mode "F2" or if I increase render_scale > 0.5.
terminate called after throwing an instance of 'c10::CUDAOutOfMemoryError' what(): CUDA out of memory. Tried to allocate 136.00 MiB (GPU 0; 3.82 GiB total capacity; 1.75 GiB already allocated; 25.06 MiB free; 1.92 GiB reserved in total by PyTorch)
nvidia-smi

[DenShlk]

It crashes after switching on neuro mode "F2" or if I increase render_scale > 0.5.

Does it render if you just rotating camera with mouse?
Gpu memory usage increases when you increase render_scale, so it is possible that there is no way to run it on your gpu with render_scale more than some particular value.

[Ilyaprok]

It renders with rotations only 3D view, Debug view and Closest Ground Truth. Neural View is black with white strip (in screenshot above).
With any render scale I watch black Neural View.

[darglein]

Hi I have a question regarding the adop_viewer. l tried the adop_viewer with pretrained models on tanks and temples dataset, it works perfectly on a laptop with a 3080 GPU (8GB VRAM) on all scenes.

I guess you are using the 40GB A100 ini files for training these models, so I also tried training my data on a GPU with similar VRAM using exact settings from train_tank_and_temples_multi.ini. However, when I attempt to visualize my checkpoints on the 3080 GPU it shows CUDA out of memory for all my scenes.

My question is: for rendering, did you make some optimizations to make the trained models fit in a GPU with less VRAM? Or is there certain configurations/arguments that allows the adop_viewer to take a model trained with high-end GPUs and render on low-end GPUs? Thanks in advance!

@qiaosongwang what resolution are your input images? For large input images we recommend to set render_scale in the dataset.ini to a value less than one. This will then be used for training and in the adop_viewer. For example, our boat scene has a render_scale of 0.5, because the input images are quiet large.

[silliconvalleypilot]

Thanks @darglein! I was using 4K images, your recommendation works. I found that I can train with dataset.ini using render_scale=1 on an A100 equivalent GPU and visualize the model by just modifying render_scale=0.5 on a 3080 GPU. All my trained scenes work now. @DenShlk @Ilyaprok I think you guys can try this out without recompiling the code.

[darglein]

@qiaosongwang You might want to set render_scale=0.5 also during training. This way the network is trained at the same scale as the inference. Depending on your scene this might give a slight improvement in quality.

[silliconvalleypilot]

@darglein Got it. Thanks!