Problem with evaluation script about cmrnet

[RaviBeagle]

Hello @cattaneod,

I solved this issue by adding the following lines to evaluate script

SETTINGS.DISCOVER_DEPENDENCIES = "none"
SETTINGS.DISCOVER_SOURCES = "none"

But now I get the following error:

python evaluate_iterative_single_CALIB.py with test_sequence=00 maps_folder=local_maps data_folder=./KITTI_ODOMETRY/sequences weight="['./checkpoints/iter1.tar','./checkpoints/iter2.tar','./checkpoints/iter3.tar']"
WARNING - CMRNet-evaluate-iterative - No observers have been added to this run
INFO - CMRNet-evaluate-iterative - Running command 'main'
INFO - CMRNet-evaluate-iterative - Started
TEST SET: Using this file: ./KITTI_ODOMETRY/sequences/test_RT_seq00_10.00_2.00.csv
ERROR - CMRNet-evaluate-iterative - Failed after 0:00:01!
Traceback (most recent calls WITHOUT Sacred internals):
  File "evaluate_iterative_single_CALIB.py", line 107, in main
    test_sequence=_config['test_sequence'])
  File "/home/sxv1kor/Temp/CMRNet/DatasetVisibilityKitti.py", line 102, in __init__
    assert len(self.test_RT) == len(self.all_files), "Something wrong with test RTs"
AssertionError: Something wrong with test RTs

Does the training produce any files used in the evaluation ?

[cattaneod]

Hi @RaviBeagle ,

the random H_init used for testing are generated the first time the evaluation is called, maybe something wrong happened when generating these transformations.

Try to remove the file ./KITTI_ODOMETRY/sequences/test_RT_seq00_10.00_2.00.csv and see if it works.

[RaviBeagle]

Try to remove the file ./KITTI_ODOMETRY/sequences/test_RT_seq00_10.00_2.00.csv and see if it works.

It does run now. But stops later:

python evaluate_iterative_single_CALIB.py with test_sequence=00 maps_folder=local_maps data_folder=./KITTI_ODOMETRY/sequences/ weight="['./checkpoints/iter1.tar','./checkpoints/iter2.tar','./checkpoints/iter3.tar']"
WARNING - CMRNet-evaluate-iterative - No observers have been added to this run
INFO - CMRNet-evaluate-iterative - Running command 'main'
INFO - CMRNet-evaluate-iterative - Started
TEST SET - Not found: ./KITTI_ODOMETRY/sequences/test_RT_seq00_10.00_2.00.csv
Generating a new one
0
0it [00:00, ?it/s]Init worker 0 with seed 273103621
Init worker 1 with seed 273103622
Init worker 4 with seed 273103625
Init worker 3 with seed 273103624
Init worker 2 with seed 273103623
Init worker 5 with seed 273103626

Iterative refinement: 
Iteration 0: 	Mean Translation Error: nan cm      Mean Rotation Error: nan °
Iteration 0: 	Median Translation Error: nan cm      Median Rotation Error: nan °

Iteration 1: 	Mean Translation Error: nan cm      Mean Rotation Error: nan °
Iteration 1: 	Median Translation Error: nan cm      Median Rotation Error: nan °

Iteration 2: 	Mean Translation Error: nan cm      Mean Rotation Error: nan °
Iteration 2: 	Median Translation Error: nan cm      Median Rotation Error: nan °

Iteration 3: 	Mean Translation Error: nan cm      Mean Rotation Error: nan °
Iteration 3: 	Median Translation Error: nan cm      Median Rotation Error: nan °

-------------------------------------------------------
Timings:
ERROR - CMRNet-evaluate-iterative - Failed after 0:00:04!
Traceback (most recent calls WITHOUT Sacred internals):
  File "evaluate_iterative_single_CALIB.py", line 431, in main
    errors_t2[i] = torch.stack(errors_t2[i])
RuntimeError: stack expects a non-empty TensorList

The evaluation script does not need the training to be run first in anycase right ?

[cattaneod]

Oh, I see your problem, can you remove the file again ./KITTI_ODOMETRY/sequences/test_RT_seq00_10.00_2.00.csv and then try evaluating with
python evaluate_iterative_single_CALIB.py with test_sequence=00 maps_folder=local_maps_0.1 data_folder=./KITTI_ODOMETRY/sequences/ weight="['./checkpoints/iter1.tar','./checkpoints/iter2.tar','./checkpoints/iter3.tar']"

[RaviBeagle]

Working now. But I get the old CUDA problem of not able to clone() on GPU (Same problem that I saw with preproces/kitti_maps.py..

Init worker 4 with seed 399312879

ERROR - CMRNet-evaluate-iterative - Failed after 0:00:04!
Traceback (most recent calls WITHOUT Sacred internals):
  File "evaluate_iterative_single_CALIB.py", line 308, in main
    rotated_point_cloud = rotate_forward(point_cloud, RT1)
  File "/home/sxv1kor/Temp/CMRNet/utils.py", line 79, in rotate_forward
    return rotate_points_torch(PC, R, T, inverse=True)
  File "/home/sxv1kor/Temp/CMRNet/utils.py", line 46, in rotate_points_torch
    RT = R.clone()
RuntimeError: CUDA error: __global__ function call is not configured
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.

[cattaneod]

I never encountered that problem, so I don't know how to help you. Have you installed cnDNN? I'm not even sure if that's required, but maybe try to install it if you didn't already.

[RaviBeagle]

Yes. Its a very strange error. I will look to PyTorch forums for help.
cuDNN is installed in my system

[RaviBeagle]

I never encountered that problem, so I don't know how to help you. Have you installed cnDNN? I'm not even sure if that's required, but maybe try to install it if you didn't already.

And now its working. I changed to Python 3.6 and CUDA 10.2 and installed the pytorch with command:
conda install pytorch torchvision torchaudio cudatoolkit=10.2 -c pytorch

And also matching cudnn as specified in the pytorch

Everything is working fine now..

[cattaneod]

And now its working. I changed to Python 3.6 and CUDA 10.2 and installed the pytorch with command: conda install pytorch torchvision torchaudio cudatoolkit=10.2 -c pytorch

And also matching cudnn as specified in the pytorch

Everything is working fine now..

I'm glad that now it's working fine!

[Hliu0313]

Thanks!