cattaneod / cmrnet Goto Github PK

python main_visibility_CALIB.py with batch_size=8 data_folder=./KITTI_ODOMETRY/ epochs=300 max_r=10 max_t=2 BASE_LEARNING_RATE=0.0001 savemodel=./checkpoints/ test_sequence=0"
Is this a PyTorch error ?
Iter 1400/1429 training loss = 1.239, time = 0.0311, time for 50 iter: 12.7947

epoch 0 total training loss = 1.323
Total epoch time = 358.35

Init worker 0 with seed 889683110
Init worker 1 with seed 889683111
Init worker 2 with seed 889683112
ERROR - CMRNet - Failed after 0:06:03!
Traceback (most recent calls WITHOUT Sacred internals):
File "main_visibility_CALIB.py", line 425, in main
depth_img = visibility.depth_image(uv, depth, depth_img, uv.shape[0], real_shape[1], real_shape[0])
RuntimeError: input_uv.is_contiguous() INTERNAL ASSERT FAILED at "src/visibility.cpp":30, please report a bug to PyTorch. input_uv must be contiguous
(cmrnet2)

In accordance with "python main_visibility_CALIB.py with batch_size=24 data_folder=./KITTI_ODOMETRY/ epochs=300 max_r=10 max_t=2 BASE_LEARNING_RATE=0.0001 savemodel=./checkpoints/ test_sequence=0"

My training results are as follows：
total test loss = 1.449
total traslation error: 164.9663721087535 cm
total rotation error: 6.641193625108441 °

This is 300-th epoch

Is there something wrong with the submitted code?

Hello @cattaneod ,

The results I have obtained from training CMRNet with KITTI type of dataset from CARLA are quite encouraging.
Now I think I could really test CMRNet as a localization module in our test vehicle online. I am in process and I think understanding the evaluate_iterative_single_CALIB.py and simplifying it is the way to go.

What I plan to do is at each instance of the simulator running at 10 FPS:

1. Get the GT pose.
2. Apply a random error to R and T to the GT pose and then
3. With just Iteration 1 of the CMRNet see how much of correction happens.

After this I could check with the same process on the vehicle.

But there seems a lot going on in evaluate_iterative_single_CALIB.py with evaluation and visualization in that code. Some hints to the most essential part of the code would be helpful..

Thanks

Hi, I noticed that the results you provide in this repo are:

While the results you provide in your paper are:

It seems like there are two different unit of translation error in repo and paper, which one is correct?

Hello, Cattaneo.

I'm trying to your this project, and it's in process of training on my PC. So, I have a question.
In the training of CMRNet, the transformation error doesn't decrease such as your paper.
Is my training correct?
I tried below command referenced as this repository README.
python main_visibility_CALIB.py with batch_size=24 data_folder=./KITTI_ODOMETRY/ epochs=300 max_r=10 max_t=2 BASE_LEARNING_RATE=0.0001 savemodel=./checkpoints/ test_sequence=0
And putting result.

I encountered an issue related to the CUDA version, so I only modified the relevant code. However, the execution results are coming out very strangely. I'm wondering if anyone else has had a similar case and if there are solutions. I can't pinpoint what the problem is, but the results obtained using a different map consistently yield almost the same values, which is suspicious. Additionally, the test loss starts from the 1st epoch and remains at 0.007. Can you provide any insights or solutions?

WARNING - CMRNet-evaluate-iterative - No observers have been added to this run
INFO - CMRNet-evaluate-iterative - Running command 'main'
INFO - CMRNet-evaluate-iterative - Started
TEST SET: Using this file: ../KITTI_ODOMETRY/sequences/test_RT_seq05_10.00_2.00.csv
801
0%| | 0/801 [00:00<?, ?it/s]Init worker 0 with seed 731988034
Init worker 1 with seed 731988035
Init worker 2 with seed 731988036
Init worker 3 with seed 731988037
Init worker 4 with seed 731988038
Init worker 5 with seed 731988039
Initial T_erorr: [0.0]
Initial R_erorr: [tensor([0.], device='cuda:0')]
100%|████████████████████████| 801/801 [00:10<00:00, 77.58it/s]
Iterative refinement:
Iteration 0: Mean Translation Error: 0.0000 cm Mean Rotation Error: 0.0000 °
Iteration 0: Median Translation Error: 0.0000 cm Median Rotation Error: 0.0000 °

Iteration 1: Mean Translation Error: 0.0996 cm Mean Rotation Error: 0.0670 °
Iteration 1: Median Translation Error: 0.0996 cm Median Rotation Error: 0.0670 °

Timings:
End!
INFO - CMRNet-evaluate-iterative - Completed after 0:00:12

(Build cuda_11.8.r11.8/compiler.31833905_0)

Hello, @cattaneod ,

I have a problem with evaluation process, while using your KITTI datasets.
I prepared every requirements and conditions, but the same error message keeps appearing like this:

with test_sequence=00 maps_folder=local_maps data_folder=./KITTI_ODOMETRY/sequen ces/ weight="['./iter1.tar', './iter2.tar', './iter3.tar']" WARNING - CMRNet-evaluate-iterative - No observers have been added to this run INFO - CMRNet-evaluate-iterative - Running command 'main' INFO - CMRNet-evaluate-iterative - Started TEST SET - Not found: ./KITTI_ODOMETRY/sequences/test_RT_seq00_10.00_2.00.csv Generating a new one 1506 0%| | 0/1506 [00:00<?, ?it/s] Init worker 0 with seed 398646958 Init worker 1 with seed 398646959 Init worker 2 with seed 398646960 Init worker 3 with seed 398646961 Init worker 4 with seed 398646962 Init worker 5 with seed 398646963 ERROR - CMRNet-evaluate-iterative - Failed after 0:04:37! Traceback (most recent calls WITHOUT Sacred internals): File "evaluate_iterative_single_CALIB.py", line 184, in main for batch_idx, sample in enumerate(tqdm(TestImgLoader)): File "/home/ius/.local/lib/python3.6/site-packages/tqdm/_tqdm.py", line 897, i n __iter__ for obj in iterable: File "/home/ius/.local/lib/python3.6/site-packages/torch/utils/data/dataloader .py", line 637, in __next__ return self._process_next_batch(batch) File "/home/ius/.local/lib/python3.6/site-packages/torch/utils/data/dataloader .py", line 658, in _process_next_batch raise batch.exc_type(batch.exc_msg) ValueError: Traceback (most recent call last): File "/home/ius/.local/lib/python3.6/site-packages/torch/utils/data/dataloader .py", line 138, in _worker_loop samples = collate_fn([dataset[i] for i in batch_indices]) File "/home/ius/.local/lib/python3.6/site-packages/torch/utils/data/dataloader .py", line 138, in <listcomp> samples = collate_fn([dataset[i] for i in batch_indices]) File "/home/ius/바탕화면/CMRNet/DatasetVisibilityKitti.py", line 196, in __get item__ R = mathutils.Euler((rotx, roty, rotz), 'XYZ') ValueError: mathutils.Euler(): invalid euler order 'XYZ'

Could you please help me with this?

Thanks for your great work and support!

Dear Author,

Thank you very much for your work, very helpful. I am very curious why the coordinate order change is used in kitti_map.py preprocessing. I suppose a classical pipeline should br to save the local_map in lidar frame(a z-up coordinate system). Then in the data_preprocessing.py, do the RT and the use velo2cam2 to transform the points in world model to camera_model(z-front). However, in your case, the coordinate order changes in kitti_map.py and camera_model. To compare, I try to add pose offset RT in lidar frame, it doesn't work, hard to converge the training. I am really afraid the work could only work in a virtual coordinate system. Please help me to clarify. Thank you so much!

Hello @cattaneod ,

We are facing some runtime error while doing training .

Could you please help us with it ?

: UserWarning: This DataLoader will create 3 worker processes in total. Our suggested max number of worker in current system is 1, which is smaller than what this DataLoader is going to create. Please be aware that excessive worker creation might get DataLoader running slow or even freeze, lower the worker number to avoid potential slowness/freeze if necessary.
cpuset_checked))
: UserWarning: To get the last learning rate computed by the scheduler, please use get_last_lr().
"please use get_last_lr().", UserWarning)
ERROR - CMRNet - Failed after 0:00:35!
Traceback (most recent calls WITHOUT Sacred internals):
File "/fs/scratch/XC_EXP_IN/CMRNet/main_visibility_CALIB.py", line 316, in main
depth_img[depth_img == 1000.] = 0.
RuntimeError: CUDA error: no kernel image is available for execution on the devi

Thanks,

Hello @cattaneod

I am checking out the code in evaluate_iterative_single_CALIB.py and some operations seem difficult to follow:

For example the line:
pc_rotated = rotate_back(pc_rotated, R, T)

Is applying the error transform to the GT point cloud.

But when I look at at another section of code:

R = quat2mat(target_rot[0])
T = tvector2mat(target_transl[0])
RT1_inv = torch.mm(T, R)
RT1 = RT1_inv.clone().inverse()

    rotated_point_cloud = rotate_forward(point_cloud, RT1)
    RTs = [RT1]

Its doing the same thing. Now the RT1 is inverted before calling rotate_forward. And within rotate_forward we have this code

if isinstance(R, torch.Tensor):
    return rotate_points_torch(PC, R, T, **inverse=True**)
else:
    return rotate_points(PC, R, T, **inverse=True**)

So its inverting again achieving same effect as the rotate_back ?

Thanks

Thanks for your work!
Maybe I missed something, but when I go through the code, in the data class DatasetVisibilityKittiSingle defined in DataVisibilityKitti.py, I didn't see that the groundtruth poses self.GTs_T and selfGTs_R are used. But if without groundtruth, how could you train the model?
Another question is, when doing inference, T_predicted, R_predicted = models[iteration](rgb, lidar) , error_t is norm of T_predicted? I thought it should be norm(T_predicted-T_groundtruth).
I guess I understand something wrong. But could you plz help me with that? Thanks a lot !

Hello @cattaneod ,

How would you recommend as the steps required to use a different data set for training and evaluating the CMRNet ?
I understand that CMRNet is scene agnostic and I could in theory evaluate the pre-trained model against a new dataset.

Thanks

Hello @cattaneod

I see that a random error_tr and error_rot can be applied to the local point cloud map as follows:

        max_angle = 10.0
        max_t = 2.0
        rotz = np.random.uniform(-max_angle, max_angle) * (3.141592 / 180.0)
        roty = np.random.uniform(-max_angle, max_angle) * (3.141592 / 180.0)
        rotx = np.random.uniform(-max_angle, max_angle) * (3.141592 / 180.0)
        transl_x = np.random.uniform(-max_t, max_t)
        transl_y = np.random.uniform(-max_t, max_t)
        transl_z = np.random.uniform(-max_t, min(max_t, 1.))

        R = mathutils.Euler((rotx, roty, rotz), 'XYZ')
        T = mathutils.Vector((transl_x, transl_y, transl_z))
        R_inv, T_inv = invert_pose(R, T)
        R = mathutils.Quaternion(R_inv)
        T = mathutils.Vector(T_inv)
        rotated_point_cloud = rotate_back(local_map.clone(), R, T)

I guess this error_tr and error_rot are with respect to local point cloud map reference frame. How can apply the same for global poses in lidar reference frame ? I guess it has something to do with changing

        transl_z = np.random.uniform(-max_t, min(max_t, 1.))

But not sure.

Thanks

Hi! I'm trying to run this project these days, and my GPU is RTX3090, CUDA version is 11.1. To meet the request of RTX3090 and CUDA11.1, I added 'arch=compute_80, code=sm_80' and 'arch=compute_80, code=compute_80' to nvcc_args, and changed cxx_args to ['-std=c++14']. However, I still get some error like this

How can i solve this error and run build this project?

Hello @cattaneod ,

I am getting an error from the sacred python module as follows:

python evaluate_iterative_single_CALIB.py with test_sequence=0 maps_folder=local_maps data_folder=./KITTI_ODOMETRY/sequences/ weight="['./checkpoints/iter1.tar','./checkpoints/iter2.tar','./checkpoints/iter3.tar']"
Traceback (most recent call last):
  File "evaluate_iterative_single_CALIB.py", line 35, in <module>
    ex = Experiment("CMRNet-evaluate-iterative")
  File "/home/sxv1kor/anaconda3/envs/cmrnet2/lib/python3.7/site-packages/sacred/experiment.py", line 80, in __init__
    _caller_globals=caller_globals)
  File "/home/sxv1kor/anaconda3/envs/cmrnet2/lib/python3.7/site-packages/sacred/ingredient.py", line 55, in __init__
    gather_sources_and_dependencies(_caller_globals, self.base_dir)
  File "/home/sxv1kor/anaconda3/envs/cmrnet2/lib/python3.7/site-packages/sacred/dependencies.py", line 487, in gather_sources_and_dependencies
    sources = gather_sources(globs, base_dir)
  File "/home/sxv1kor/anaconda3/envs/cmrnet2/lib/python3.7/site-packages/sacred/dependencies.py", line 438, in get_sources_from_imported_modules
    return get_sources_from_modules(iterate_imported_modules(globs), base_path)
  File "/home/sxv1kor/anaconda3/envs/cmrnet2/lib/python3.7/site-packages/sacred/dependencies.py", line 407, in get_sources_from_modules
    filename = os.path.abspath(mod.__file__)
  File "/home/sxv1kor/anaconda3/envs/cmrnet2/lib/python3.7/posixpath.py", line 378, in abspath
    path = os.fspath(path)
TypeError: expected str, bytes or os.PathLike object, not NoneType

Any idea how to to solve this issue ? Is this related to Python 3.7 ?

Also one other problem I saw when running the training script was that it expected the poses.csv file for each training sequence to be be in its folder within KITTI_ODOMETRY. I manually copied kitti-(nn).csv files from the data folder to its corresponding place.

Thanks

Hello @cattaneod,

I just wanted to know if there is anything specific with KITTI extrinsics (Tr) values ?

For example in sequence 00 of KITTI the Tr value is:

4.276802385584e-04 -9.999672484946e-01 -8.084491683471e-03 -1.198459927713e-02 
-7.210626507497e-03 8.081198471645e-03 -9.999413164504e-01 -5.403984729748e-02 
9.999738645903e-01 4.859485810390e-04 -7.206933692422e-03 -2.921968648686e-01

A I understand this is the homogeneous transform from velodyne frame to cam0 frame. But what I do not understand is why the translation values have the signs flipped. I mean the values (-1.198459927713e-02 and -5.403984729748e-02 and -2.921968648686e-01). The pykitti is finally generating the velo2cam2 matrix as:

[[ 4.2768e-04, -9.9997e-01, -8.0845e-03,  5.1147e-02],
        [-7.2106e-03,  8.0812e-03, -9.9994e-01, -5.4040e-02],
        [ 9.9997e-01,  4.8595e-04, -7.2069e-03, -2.9220e-01],
        [ 0.0000e+00,  0.0000e+00,  0.0000e+00,  1.0000e+00]]

In my CARLA generated (exact simulated world( transforms the Tr matrix for cam0 is (which I add to generated calib.txt)

[[ 0.0000, -1.0000, 0.0000, 0.0000],
[ 0.0000, 0.0000, -1.0000, 0.0800],
[ 1.0000, 0.0000, 0.0000, 0.2700],
[ 0.0000, 0.0000, 0.0000, 1.0000]

And for cam2:

[[ 0.0000, -1.0000, 0.0000, -0.0600],
[ 0.0000, 0.0000, -1.0000, 0.0800],
[ 1.0000, 0.0000, 0.0000, 0.2700],
[ 0.0000, 0.0000, 0.0000, 1.0000]

which seems correct have the correct translation values in the camera reference frame. Also in my visualization of GT with evaluate script I see a lateral shift between camera pixels and lidar image.

Thank you

Hello,

I am following the README.md and trying to generate the local maps with the given command:

python preprocess/kitti_maps.py --sequence 00 --kitti_folder ./KITTI_ODOMETRY/
Sequnce: 00
Traceback (most recent call last):
File "preprocess/kitti_maps.py", line 54, in
kitti = pykitti.odometry(args.kitti_folder, sequence)
File "/home/sxv1kor/anaconda3/envs/cmrnet/lib/python3.6/site-packages/pykitti/odometry.py", line 33, in init
self._load_calib()
File "/home/sxv1kor/anaconda3/envs/cmrnet/lib/python3.6/site-packages/pykitti/odometry.py", line 169, in _load_calib
data['T_cam0_velo'] = np.reshape(filedata['Tr'], (3, 4))
KeyError: 'Tr'

All requirements of python environment are satisfied and I can see this line:

Requirement already satisfied: pykitti in /home/sxv1kor/anaconda3/envs/cmrnet/lib/python3.6/site-packages (from -r requirements.txt (line 11)) (0.3.1)

Any hints to solve this issue ? would be helpful.

KITTI data set was downloaded from: https://s3.eu-central-1.amazonaws.com/avg-kitti/data_odometry_color.zip

Thanks

I have a question regarding the occlusion filtering. In the training part of CMRNet, there is no maxpooling done to filter out points behind other points, since this is done through this cone creation and filtering implemented in the visibility package, from my understanding. My question now is, why is there a maxpooling applied when we want to visualize the rgb and lidar points overlaid, as in

which is used here

Thank you for your great repository and support!

Hello @cattaneod ,

What does these lines of code do:

                if dir == test_sequence and split.startswith('test'):
                    self.all_files.append(os.path.join(dir, str(row['timestamp'])))
                elif (not dir == test_sequence) and split == 'train':
                    self.all_files.append(os.path.join(dir, str(row['timestamp'])))

I think something else was meant, but the all_files list is appended whether its test set or train set ?

Also in the same file I dont see camera model self.model = CameraModel()
used anywhere in that file.

The training on my NVIDIA A3000 GPU fails after Epoch 0. I used batch size 8 since I was getting out of GPU memory error. System: Ubuntu 20.04.
Versions:
torch 1.12.0.dev20220520+cu116 pypi_0 pypi
torchvision 0.13.0.dev20220520+cu116 pypi_0 pypi
cupy-cuda116 10.4.0 pypi_0 pypi
"python main_visibility_CALIB.py with batch_size=8 data_folder=./KITTI_ODOMETRY/ epochs=300 max_r=10 max_t=2 BASE_LEARNING_RATE=0.0001 savemodel=./checkpoints/ test_sequence=0"
Is this a PyTorch error ?
Iter 1400/1429 training loss = 1.239, time = 0.0311, time for 50 iter: 12.7947

epoch 0 total training loss = 1.323
Total epoch time = 358.35

Init worker 0 with seed 889683110
Init worker 1 with seed 889683111
Init worker 2 with seed 889683112
ERROR - CMRNet - Failed after 0:06:03!
Traceback (most recent calls WITHOUT Sacred internals):
File "main_visibility_CALIB.py", line 425, in main
depth_img = visibility.depth_image(uv, depth, depth_img, uv.shape[0], real_shape[1], real_shape[0])
RuntimeError: input_uv.is_contiguous() INTERNAL ASSERT FAILED at "src/visibility.cpp":30, please report a bug to PyTorch. input_uv must be contiguous
(cmrnet2)

Hi, thanks for your insteresting work! I have evaluated your provided pretrained weights, but the performances cannot be reproduced. I got the performance as:

, which is much lower than your performance. I guess it may be caused by different ground-truth pose during evaluation? So, could you provide your test_RT_seq00__.csv?
Thanks at advance.

When i run kitti_maps.py, it displays "killed", and fails to create .h5 files. How to solve this problem.

I just wanted some clarification on the conventions. Great work by the way!
These ground-truth poses are given in what frame? Are they the "world" frame used in the kitti dataset?.
Additionally, are they the poses of lidar or the camera?

I tried checking if there was a constant transformation between the kitti ground-truth poses and the ones uploaded here, but there does not seem to be one. Hence I am confused.

Does CMRNet++ plan to open source? Thank you for your answer.

Hi, can you share a pretrained model that help me do a quick verification on this approach?

Will later versions of pytorch (say 1.4 and up) be compatible with the CMRNet code? Thanks