Problem with LocalMaps Generation about cmrnet

[cattaneod]

You probably miss the KITTI calibration data: https://s3.eu-central-1.amazonaws.com/avg-kitti/data_odometry_calib.zip

[RaviBeagle]

As far as I can see the KITTI data set sequence folders each have a calib.txt and times.txt already. Should I overwrite them with the data from the ones in https://s3.eu-central-1.amazonaws.com/avg-kitti/data_odometry_calib.zip ?

[cattaneod]

Yes, the ones that are already there needs to be overwritten with the ones in data_odometry_calib.zip

[RaviBeagle]

Thank you. There seems one minor issue in the scripts. The kitty_maps.py script requires the sequences to be in the folder './KITTI_ODOMETRY/sequences/'

The training and evaluation script requires the sequences to be in ./KITTI_ODOMETRY/ folder itself.

[RaviBeagle]

Yes, the ones that are already there needs to be overwritten with the ones in data_odometry_calib.zip

Thank you very much. Its working now.
May I know if the current repository is of CMRNet or it already includes the CMRNet++ ? I did not see any references to the RANSAC and PnP algorithms..

[RaviBeagle]

Also I would like to know if there is any hard dependency on a version of PyTorch.
Because I am using a newer NVIDIA GPU and still getting a CUDA error:

Traceback (most recent call last):
File "preprocess/kitti_maps.py", line 108, in
local_map = voxelized.clone()
RuntimeError: CUDA error: no kernel image is available for execution on the device

I see some discussion on the issue : #7
But I am not sure of the changes which needs to be done..

nvidia-smi
Fri May 20 10:55:13 2022
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 510.73.05 Driver Version: 510.73.05 CUDA Version: 11.6 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 NVIDIA RTX A300... Off | 00000000:01:00.0 On | N/A |
| N/A 42C P8 14W / N/A | 588MiB / 6144MiB | 1% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+

[cattaneod]

Thank you very much. Its working now. May I know if the current repository is of CMRNet or it already includes the CMRNet++ ? I did not see any references to the RANSAC and PnP algorithms..

This repository only includes CMRNet

Also I would like to know if there is any hard dependency on a version of PyTorch. Because I am using a newer NVIDIA GPU and still getting a CUDA error:

Traceback (most recent call last): File "preprocess/kitti_maps.py", line 108, in local_map = voxelized.clone() RuntimeError: CUDA error: no kernel image is available for execution on the device

I see some discussion on the issue : #7 But I am not sure of the changes which needs to be done..

nvidia-smi Fri May 20 10:55:13 2022 +-----------------------------------------------------------------------------+ | NVIDIA-SMI 510.73.05 Driver Version: 510.73.05 CUDA Version: 11.6 | |-------------------------------+----------------------+----------------------+ | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |===============================+======================+======================| | 0 NVIDIA RTX A300... Off | 00000000:01:00.0 On | N/A | | N/A 42C P8 14W / N/A | 588MiB / 6144MiB | 1% Default | | | | N/A | +-------------------------------+----------------------+----------------------+

Which pytorch version are you using? according to pytorch/pytorch#49161, nvidia 3090 is only supported starting from pytorch 1.7.1

Yesterday I updated this repository, and now it should work with any version of pytorch (I tested with 1.10)

[RaviBeagle]

I am using pip show torch
Name: torch
Version: 1.0.1.post2

I have updated to the latest repository and I ran pip install -r requirements.txt again.
And I also just noticed that https://github.com/cattaneod/CMRNet documentation there is an additional step that is not mentioned in the README.md file:
viz.
_ "And finally, install the correlation_cuda and the visibility package:

cd models/CMRNet/correlation_package/
python setup.py install
cd ../../..
python setyp.py install" _

So I have done this step. And I also enabled : '-gencode', 'arch=compute_80,code=sm_80',

Now I see a different error actually:

_ THCudaCheck FAIL file=/pytorch/aten/src/THC/THCGeneral.cpp line=51 error=30 : unknown error
Traceback (most recent call last):
File "preprocess/kitti_maps.py", line 94, in
voxelized = voxelized.to(args.device)
File "/home/sxv1kor/anaconda3/envs/cmrnet/lib/python3.6/site-packages/torch/cuda/init.py", line 162, in _lazy_init
torch._C._cuda_init()
RuntimeError: cuda runtime error (30) : unknown error at /pytorch/aten/src/THC/THCGeneral.cpp:51 _

[cattaneod]

You need to update pytorch to a newer version to run it with nvidia 3090.

I would suggest you to create a new conda environment and follow the new installation instructions from scratch. I'm sorry for your inconvenience.

PS: I would also suggest to clone the repository again, as you might have some old built files, and python would use them instead of build them again. Or, alternatively, remove any build, dist folder and any *.so file from the root of the repository and from models/CMRNet/correlation_package

[RaviBeagle]

You need to update pytorch to a newer version to run it with nvidia 3090.

I would suggest you to create a new conda environment and follow the new installation instructions from scratch. I'm sorry for your inconvenience.

PS: I would also suggest to clone the repository again, as you might have some old built files, and python would use them instead of build them again. Or, alternatively, remove any build, dist folder and any *.so file from the root of the repository and from models/CMRNet/correlation_package

No problem! I am doing just that now and I have to install torch, torchvision and cupy.
Is it fine if I install cupy-cuda102 ?
I actually have RTX A3000 which is similar to RTX 3060..

[cattaneod]

No problem! I am doing just that now and I have to install torch, torchvision and cupy. Is it fine if I install cupy-cuda102 ? I actually have RTX A3000 which is similar to RTX 3060..

No, you need to use the same cuda version for everything (pytorch, cupy, and so on), and it should be the version you have installed in your system. As the AMPERE architecture require CUDA 11.x, cupy-cuda102 won't work.

EDIT: you should check your cuda version with nvcc --version, and use that exact same version to install pytorch, torchvision and cupy

[RaviBeagle]

So, all right the commands should do the trick

nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2019 NVIDIA Corporation
Built on Sun_Jul_28_19:07:16_PDT_2019
Cuda compilation tools, release 10.1, V10.1.243

pip install cupy_cuda101=9.6.1
pip install torch==1.5.0+cu101 torchvision==0.6.0+cu101 -f

[cattaneod]

I don't think you can use cuda 10.1 with your nvidia A3000, you can try, but if it doesn't work, you'll need to update cuda to (at least) 11

https://docs.nvidia.com/cuda/ampere-compatibility-guide/index.html

CUDA applications built using CUDA Toolkit versions 2.1 through 10.2 are compatible with NVIDIA Ampere architecture based GPUs as long as they are built to include PTX versions of their kernels.

I can't guarantee that it will works with cuda 10.1

[RaviBeagle]

I don't think you can use cuda 10.1 with your nvidia A3000, you can try, but if it doesn't work, you'll need to update cuda to (at least) 11

https://docs.nvidia.com/cuda/ampere-compatibility-guide/index.html

CUDA applications built using CUDA Toolkit versions 2.1 through 10.2 are compatible with NVIDIA Ampere architecture based GPUs as long as they are built to include PTX versions of their kernels.

I can't guarantee that it will works with cuda 10.1

Now I have updated the CUDA toolkit:
Device 0: "NVIDIA RTX A3000 Laptop GPU"
CUDA Driver Version / Runtime Version 11.6 / 11.6

I am using Ubuntu 20.04 and I had to use a conda 3.7 environment because the CuPy requirement(https://docs.cupy.dev/en/stable/install.html).
For PyTorch: https://discuss.pytorch.org/t/pytorch-cuda-11-6/149647/2
Now the versions installed are:
Name Version Build Channel
torch 1.12.0.dev20220520+cu116 pypi_0 pypi
torchvision 0.13.0.dev20220520+cu116 pypi_0 pypi
cupy-cuda116 10.4.0 pypi_0 pypi

And these are working fine now:
cd models/CMRNet/correlation_package/
python setup.py install
cd ../../..
python setyp.py install

After this the command
python preprocess/kitti_maps.py --sequence 00 --kitti_folder ./KITTI_ODOMETRY/

gives error :

voxelized = torch.tensor(downpcd.points, dtype=torch.float)
0%| | 0/4541 [00:00<?, ?it/s]
Traceback (most recent call last):
File "preprocess/kitti_maps.py", line 108, in
local_map = voxelized.clone()
RuntimeError: CUDA error: global function call is not configured
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.

[RaviBeagle]

Hello @cattaneod

I solved this problem by switching to CPU computation instead of GPU. The ticket can be closed.

[cattaneod]

I don't think it will work with cpu, some functions are implemented only in GPU.

To me it seems that your ptoblem is still cuda. Check the version with nvcc, not with nvidia-smi, as the latter will show the supported cuda version, not the installed version.

[RaviBeagle]

They are matching:

nvidia-smi
Wed May 25 19:22:14 2022
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 510.73.05 Driver Version: 510.73.05 CUDA Version: 11.6

nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2021 NVIDIA Corporation
Built on Fri_Dec_17_18:16:03_PST_2021
Cuda compilation tools, release 11.6, V11.6.55
Build cuda_11.6.r11.6/compiler.30794723_0PU

I also confirmed it with deviceQuery (from cuda-samples):

/cuda-samples/Samples/1_Utilities/deviceQuery$ ./deviceQuery
./deviceQuery Starting...

CUDA Device Query (Runtime API) version (CUDART static linking)

Detected 1 CUDA Capable device(s)

Device 0: "NVIDIA RTX A3000 Laptop GPU"
CUDA Driver Version / Runtime Version 11.6 / 11.6

Its only the kitti_maps.py that I switched from GPU to CPU (it was fairly simple) and its working fine. The learning is still with GPU.

In the kitti_maps.py:
voxelized and poses are transferred to device memory. vox_intensity is CPU memory. But the problem seems to be with
local_map = voxelized.clone(). Something with the objects between CPU memory and GPU memory might be causing the issue/

[RaviBeagle]

This repository only includes CMRNet

Is there a plan to also put CMRNet++ on github ?

[RaviBeagle]

I have been able to run the preprocess\kitti_maps.py on GPU with some changes.

voxelized = voxelized.to(args.device)
vox_intensity = torch.tensor(downpcd.colors, dtype=torch.float)[:, 0:1].t().to(args.device)

# veclo2cam2 is in GPU as a constant
velo2cam2 = torch.from_numpy(kitti.calib.T_cam2_velo).float().to(args.device).to(args.device)

if True:
    # SAVE SINGLE PCs
    if not os.path.exists(os.path.join(args.kitti_folder, 'sequences', sequence,
                                    f'local_maps_{args.voxel_size}')):
        os.mkdir(os.path.join(args.kitti_folder, 'sequences', sequence, f'local_maps_{args.voxel_size}'))
    for i in tqdm(range(first_frame, last_frame)):
        pose = poses[i]
        pose = pose.to(args.device)
        pose = pose.inverse()

        #local_map = voxelized.to(args.device)
        #local_intensity = vox_intensity.to(args.device)
        local_map = torch.mm(pose, voxelized).t()
        indexes = local_map[:, 1] > -25.
        indexes = indexes & (local_map[:, 1] < 25.)
        indexes = indexes & (local_map[:, 0] > -10.)
        indexes = indexes & (local_map[:, 0] < 100.)
        local_map = local_map[indexes]
        local_intensity = vox_intensity[:, indexes]

[RaviBeagle]

I think there might be possibility of using CUDA 10.2 on Ampere Architecture. Will explore it:
https://docs.nvidia.com/cuda/ampere-compatibility-guide/index.html#verifying-ampere-compatibility-using-cuda-10-2