Comments (20)
You probably miss the KITTI calibration data: https://s3.eu-central-1.amazonaws.com/avg-kitti/data_odometry_calib.zip
from cmrnet.
As far as I can see the KITTI data set sequence folders each have a calib.txt and times.txt already. Should I overwrite them with the data from the ones in https://s3.eu-central-1.amazonaws.com/avg-kitti/data_odometry_calib.zip ?
from cmrnet.
Yes, the ones that are already there needs to be overwritten with the ones in data_odometry_calib.zip
from cmrnet.
Thank you. There seems one minor issue in the scripts. The kitty_maps.py script requires the sequences to be in the folder './KITTI_ODOMETRY/sequences/'
The training and evaluation script requires the sequences to be in ./KITTI_ODOMETRY/ folder itself.
from cmrnet.
Yes, the ones that are already there needs to be overwritten with the ones in data_odometry_calib.zip
Thank you very much. Its working now.
May I know if the current repository is of CMRNet or it already includes the CMRNet++ ? I did not see any references to the RANSAC and PnP algorithms..
from cmrnet.
Also I would like to know if there is any hard dependency on a version of PyTorch.
Because I am using a newer NVIDIA GPU and still getting a CUDA error:
Traceback (most recent call last):
File "preprocess/kitti_maps.py", line 108, in
local_map = voxelized.clone()
RuntimeError: CUDA error: no kernel image is available for execution on the device
I see some discussion on the issue : #7
But I am not sure of the changes which needs to be done..
nvidia-smi
Fri May 20 10:55:13 2022
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 510.73.05 Driver Version: 510.73.05 CUDA Version: 11.6 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 NVIDIA RTX A300... Off | 00000000:01:00.0 On | N/A |
| N/A 42C P8 14W / N/A | 588MiB / 6144MiB | 1% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
from cmrnet.
Thank you very much. Its working now. May I know if the current repository is of CMRNet or it already includes the CMRNet++ ? I did not see any references to the RANSAC and PnP algorithms..
This repository only includes CMRNet
Also I would like to know if there is any hard dependency on a version of PyTorch. Because I am using a newer NVIDIA GPU and still getting a CUDA error:
Traceback (most recent call last): File "preprocess/kitti_maps.py", line 108, in local_map = voxelized.clone() RuntimeError: CUDA error: no kernel image is available for execution on the device
I see some discussion on the issue : #7 But I am not sure of the changes which needs to be done..
nvidia-smi Fri May 20 10:55:13 2022 +-----------------------------------------------------------------------------+ | NVIDIA-SMI 510.73.05 Driver Version: 510.73.05 CUDA Version: 11.6 | |-------------------------------+----------------------+----------------------+ | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |===============================+======================+======================| | 0 NVIDIA RTX A300... Off | 00000000:01:00.0 On | N/A | | N/A 42C P8 14W / N/A | 588MiB / 6144MiB | 1% Default | | | | N/A | +-------------------------------+----------------------+----------------------+
Which pytorch version are you using? according to pytorch/pytorch#49161, nvidia 3090 is only supported starting from pytorch 1.7.1
Yesterday I updated this repository, and now it should work with any version of pytorch (I tested with 1.10)
from cmrnet.
I am using pip show torch
Name: torch
Version: 1.0.1.post2
I have updated to the latest repository and I ran pip install -r requirements.txt again.
And I also just noticed that https://github.com/cattaneod/CMRNet documentation there is an additional step that is not mentioned in the README.md file:
viz.
_ "And finally, install the correlation_cuda and the visibility package:
cd models/CMRNet/correlation_package/
python setup.py install
cd ../../..
python setyp.py install" _
So I have done this step. And I also enabled : '-gencode', 'arch=compute_80,code=sm_80',
Now I see a different error actually:
_ THCudaCheck FAIL file=/pytorch/aten/src/THC/THCGeneral.cpp line=51 error=30 : unknown error
Traceback (most recent call last):
File "preprocess/kitti_maps.py", line 94, in
voxelized = voxelized.to(args.device)
File "/home/sxv1kor/anaconda3/envs/cmrnet/lib/python3.6/site-packages/torch/cuda/init.py", line 162, in _lazy_init
torch._C._cuda_init()
RuntimeError: cuda runtime error (30) : unknown error at /pytorch/aten/src/THC/THCGeneral.cpp:51 _
from cmrnet.
You need to update pytorch to a newer version to run it with nvidia 3090.
I would suggest you to create a new conda environment and follow the new installation instructions from scratch. I'm sorry for your inconvenience.
PS: I would also suggest to clone the repository again, as you might have some old built files, and python would use them instead of build them again. Or, alternatively, remove any build
, dist
folder and any *.so
file from the root of the repository and from models/CMRNet/correlation_package
from cmrnet.
You need to update pytorch to a newer version to run it with nvidia 3090.
I would suggest you to create a new conda environment and follow the new installation instructions from scratch. I'm sorry for your inconvenience.
PS: I would also suggest to clone the repository again, as you might have some old built files, and python would use them instead of build them again. Or, alternatively, remove any
build
,dist
folder and any*.so
file from the root of the repository and frommodels/CMRNet/correlation_package
No problem! I am doing just that now and I have to install torch, torchvision and cupy.
Is it fine if I install cupy-cuda102 ?
I actually have RTX A3000 which is similar to RTX 3060..
from cmrnet.
No problem! I am doing just that now and I have to install torch, torchvision and cupy. Is it fine if I install cupy-cuda102 ? I actually have RTX A3000 which is similar to RTX 3060..
No, you need to use the same cuda version for everything (pytorch, cupy, and so on), and it should be the version you have installed in your system. As the AMPERE architecture require CUDA 11.x, cupy-cuda102 won't work.
EDIT: you should check your cuda version with nvcc --version
, and use that exact same version to install pytorch, torchvision and cupy
from cmrnet.
So, all right the commands should do the trick
nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2019 NVIDIA Corporation
Built on Sun_Jul_28_19:07:16_PDT_2019
Cuda compilation tools, release 10.1, V10.1.243
pip install cupy_cuda101=9.6.1
pip install torch==1.5.0+cu101 torchvision==0.6.0+cu101 -f
from cmrnet.
I don't think you can use cuda 10.1 with your nvidia A3000, you can try, but if it doesn't work, you'll need to update cuda to (at least) 11
https://docs.nvidia.com/cuda/ampere-compatibility-guide/index.html
CUDA applications built using CUDA Toolkit versions 2.1 through 10.2 are compatible with NVIDIA Ampere architecture based GPUs as long as they are built to include PTX versions of their kernels.
I can't guarantee that it will works with cuda 10.1
from cmrnet.
I don't think you can use cuda 10.1 with your nvidia A3000, you can try, but if it doesn't work, you'll need to update cuda to (at least) 11
https://docs.nvidia.com/cuda/ampere-compatibility-guide/index.html
CUDA applications built using CUDA Toolkit versions 2.1 through 10.2 are compatible with NVIDIA Ampere architecture based GPUs as long as they are built to include PTX versions of their kernels.
I can't guarantee that it will works with cuda 10.1
Now I have updated the CUDA toolkit:
Device 0: "NVIDIA RTX A3000 Laptop GPU"
CUDA Driver Version / Runtime Version 11.6 / 11.6
I am using Ubuntu 20.04 and I had to use a conda 3.7 environment because the CuPy requirement(https://docs.cupy.dev/en/stable/install.html).
For PyTorch: https://discuss.pytorch.org/t/pytorch-cuda-11-6/149647/2
Now the versions installed are:
Name Version Build Channel
torch 1.12.0.dev20220520+cu116 pypi_0 pypi
torchvision 0.13.0.dev20220520+cu116 pypi_0 pypi
cupy-cuda116 10.4.0 pypi_0 pypi
And these are working fine now:
cd models/CMRNet/correlation_package/
python setup.py install
cd ../../..
python setyp.py install
After this the command
python preprocess/kitti_maps.py --sequence 00 --kitti_folder ./KITTI_ODOMETRY/
gives error :
voxelized = torch.tensor(downpcd.points, dtype=torch.float)
0%| | 0/4541 [00:00<?, ?it/s]
Traceback (most recent call last):
File "preprocess/kitti_maps.py", line 108, in
local_map = voxelized.clone()
RuntimeError: CUDA error: global function call is not configured
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
from cmrnet.
Hello @cattaneod
I solved this problem by switching to CPU computation instead of GPU. The ticket can be closed.
from cmrnet.
I don't think it will work with cpu, some functions are implemented only in GPU.
To me it seems that your ptoblem is still cuda. Check the version with nvcc, not with nvidia-smi
, as the latter will show the supported cuda version, not the installed version.
from cmrnet.
They are matching:
nvidia-smi
Wed May 25 19:22:14 2022
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 510.73.05 Driver Version: 510.73.05 CUDA Version: 11.6
nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2021 NVIDIA Corporation
Built on Fri_Dec_17_18:16:03_PST_2021
Cuda compilation tools, release 11.6, V11.6.55
Build cuda_11.6.r11.6/compiler.30794723_0PU
I also confirmed it with deviceQuery (from cuda-samples):
/cuda-samples/Samples/1_Utilities/deviceQuery$ ./deviceQuery
./deviceQuery Starting...
CUDA Device Query (Runtime API) version (CUDART static linking)
Detected 1 CUDA Capable device(s)
Device 0: "NVIDIA RTX A3000 Laptop GPU"
CUDA Driver Version / Runtime Version 11.6 / 11.6
Its only the kitti_maps.py that I switched from GPU to CPU (it was fairly simple) and its working fine. The learning is still with GPU.
In the kitti_maps.py:
voxelized and poses are transferred to device memory. vox_intensity is CPU memory. But the problem seems to be with
local_map = voxelized.clone(). Something with the objects between CPU memory and GPU memory might be causing the issue/
from cmrnet.
This repository only includes CMRNet
Is there a plan to also put CMRNet++ on github ?
from cmrnet.
I have been able to run the preprocess\kitti_maps.py on GPU with some changes.
voxelized = voxelized.to(args.device) vox_intensity = torch.tensor(downpcd.colors, dtype=torch.float)[:, 0:1].t().to(args.device) # veclo2cam2 is in GPU as a constant velo2cam2 = torch.from_numpy(kitti.calib.T_cam2_velo).float().to(args.device).to(args.device) if True: # SAVE SINGLE PCs if not os.path.exists(os.path.join(args.kitti_folder, 'sequences', sequence, f'local_maps_{args.voxel_size}')): os.mkdir(os.path.join(args.kitti_folder, 'sequences', sequence, f'local_maps_{args.voxel_size}')) for i in tqdm(range(first_frame, last_frame)): pose = poses[i] pose = pose.to(args.device) pose = pose.inverse() #local_map = voxelized.to(args.device) #local_intensity = vox_intensity.to(args.device) local_map = torch.mm(pose, voxelized).t() indexes = local_map[:, 1] > -25. indexes = indexes & (local_map[:, 1] < 25.) indexes = indexes & (local_map[:, 0] > -10.) indexes = indexes & (local_map[:, 0] < 100.) local_map = local_map[indexes] local_intensity = vox_intensity[:, indexes]
from cmrnet.
I think there might be possibility of using CUDA 10.2 on Ampere Architecture. Will explore it:
https://docs.nvidia.com/cuda/ampere-compatibility-guide/index.html#verifying-ampere-compatibility-using-cuda-10-2
from cmrnet.
Related Issues (20)
- Training fails after epoch 0 HOT 2
- Problem with evaluation script HOT 10
- About model performance HOT 3
- Using a different data set for training and evaluation HOT 12
- RuntimeError: CUDA error: no kernel image is available for execution HOT 1
- Some hints needed on converting to online use HOT 10
- Line of code seems incorrect HOT 1
- Some operations seem difficult to follow HOT 3
- How can I apply the error_tr and error_rot for pose ? HOT 2
- Is there any specifc format for the KITTI extrinsics ?
- Ground truth test pose is required HOT 1
- Occlusion filtering (train vs eval show) HOT 2
- ValueError: mathutils.Euler(): invalid euler order 'XYZ' HOT 2
- When i run kitti_maps.py, it displays "killed". HOT 3
- Does CMRNet++ plan to open source? HOT 1
- kitti_map.py axis order issue HOT 7
- Evaluated Error Result issue
- How can i run this project on RTX3090? HOT 13
- Training fails after Epoch 0 HOT 6
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from cmrnet.