Error in reading ground-truth labels while evaluating pre-trained model about smoke

[ccerhan]

Thanks a lot for making your code open source, and for fostering further research in this field!
I downloaded the pre-trained model that is linked here, and I also used the ImageSets as described in the issue #3 and placed the same in both the training and testing data folders.
On running only the evaluation on the pre-trained model, using the kitti eval folder provided here as referenced in issue #4 I got this error:-
ERROR: Couldn't read: 004627.txt of ground truth. Please write me an email! An error occurred while processing your results
Could you please tell me how this error came about? And how I can fix it?

The evaluation will run the command "./evaluate_object_3d_offline {} {}".format(label_dir, output_dir)" in kitti_eval.py. But the default config "smoke_gn_vector.yaml" using " TEST: ("kitti_test",) " and TEST_SPLIT: "test" . And the test dataset has no label file, thus causing this problem. So you need to replace it with "TEST: ("kitti_train",)" and TEST_SPLIT: "val". Note that you should make sure the label_dir is exists and "./evaluate_object_3d_offline" can find it.

To make label_dir that can be found by "./evaluate_object_3d_offline", change the line where label_dir is set to label_dir = os.path.join('..', '..', '..', '..', '..', '..', getattr(dataset, 'label_dir')).

[EphChem]

Thanks a lot for making your code open source, and for fostering further research in this field!
I downloaded the pre-trained model that is linked here, and I also used the ImageSets as described in the issue #3 and placed the same in both the training and testing data folders.
On running only the evaluation on the pre-trained model, using the kitti eval folder provided here as referenced in issue #4 I got this error:-
ERROR: Couldn't read: 004627.txt of ground truth. Please write me an email! An error occurred while processing your results
Could you please tell me how this error came about? And how I can fix it?

The evaluation will run the command "./evaluate_object_3d_offline {} {}".format(label_dir, output_dir)" in kitti_eval.py. But the default config "smoke_gn_vector.yaml" using " TEST: ("kitti_test",) " and TEST_SPLIT: "test" . And the test dataset has no label file, thus causing this problem. So you need to replace it with "TEST: ("kitti_train",)" and TEST_SPLIT: "val". Note that you should make sure the label_dir is exists and "./evaluate_object_3d_offline" can find it.

To make label_dir that can be found by "./evaluate_object_3d_offline", change the line where label_dir is set to label_dir = os.path.join('..', '..', '..', '..', '..', '..', getattr(dataset, 'label_dir')).

On top of what @ccerhan mentioned above, I had to:

change os.chdir('../smoke/data/datasets/evaluation/kitti/kitti_eval') to os.chdir('./smoke/data/datasets/evaluation/kitti/kitti_eval')

change os.chdir('../tools') to os.chdir('../../../../../../tools')

also make sure to have gnuplot, ghostscript, texlive-extra-utils installed

[gaoyuchris]

Thanks a lot for making your code open source, and for fostering further research in this field!

I downloaded the pre-trained model that is linked here, and I also used the ImageSets as described in the issue #3 and placed the same in both the training and testing data folders.
On running only the evaluation on the pre-trained model, using the kitti eval folder provided here as referenced in issue #4 I got this error:-

ERROR: Couldn't read: 004627.txt of ground truth. Please write me an email! An error occurred while processing your results

Could you please tell me how this error came about? And how I can fix it?

The evaluation will run the command "./evaluate_object_3d_offline {} {}".format(label_dir, output_dir)" in kitti_eval.py. But the default config "smoke_gn_vector.yaml" using " TEST: ("kitti_test",) " and TEST_SPLIT: "test" . And the test dataset has no label file, thus causing this problem. So you need to replace it with "TEST: ("kitti_train",)" and TEST_SPLIT: "val". Note that you should make sure the label_dir is exists and "./evaluate_object_3d_offline" can find it.

[bijonguha]

I also faced the same problem. I solved it using following steps :

I noticed that predictions were getiing generated in my case in /content/SMOKE/tools/logs/inference/kitti_train/data. But evaluation was throwing error :

number of files for evaluation: 3769
ERROR: Couldn't read: 001537.txt of ground truth. Please write me an email!
An error occured while processing your results.
Traceback (most recent call last):
File "tools/plain_train_net.py", line 100, in
args=(args,),
File "/content/SMOKE/smoke/engine/launch.py", line 56, in launch
main_func(*args)
File "tools/plain_train_net.py", line 79, in main
return run_test(cfg, model)
File "/content/SMOKE/smoke/engine/test_net.py", line 26, in run_test
output_folder=output_folder,
File "/content/SMOKE/smoke/engine/inference.py", line 74, in inference
output_folder=output_folder, )
File "/content/SMOKE/smoke/data/datasets/evaluation/init.py", line 26, in evaluate
return kitti_evaluation(**args)
File "/content/SMOKE/smoke/data/datasets/evaluation/kitti/kitti_eval.py", line 28, in kitti_evaluation
logger=logger
File "/content/SMOKE/smoke/data/datasets/evaluation/kitti/kitti_eval.py", line 55, in do_kitti_detection_evaluation
os.chdir('../tools')
FileNotFoundError: [Errno 2] No such file or directory: '../tools'

So I used following repo for evaluation and result generation: https://github.com/asharakeh/kitti_native_evaluation.git

'apt-get install texlive-extra-utils'
'apt-get install gnuplot'
'apt-get install ghostscript'

'git clone https://github.com/asharakeh/kitti_native_evaluation.git'

'cd /content/kitti_native_evaluation'
'cmake ./'
'make'

'./evaluate_object_3d_offline /content/SMOKE/datasets/kitti/training/label_2 /content/SMOKE/tools/logs/inference/kitti_train'

result would be similar to this :
PDFCROP 1.38, 2012/11/02 - Copyright (c) 2002-2012 by Heiko Oberdiek.
==> 1 page written on car_detection_AP.pdf'. car_orientation_AOS : 99.565636 96.361633 96.095497 PDFCROP 1.38, 2012/11/02 - Copyright (c) 2002-2012 by Heiko Oberdiek. ==> 1 page written on car_orientation_AOS.pdf'.
pedestrian_detection_AP : 74.489685 72.431320 66.264664
PDFCROP 1.38, 2012/11/02 - Copyright (c) 2002-2012 by Heiko Oberdiek.
==> 1 page written on pedestrian_detection_AP.pdf'. pedestrian_orientation_AOS : 70.911919 67.916069 62.048210 PDFCROP 1.38, 2012/11/02 - Copyright (c) 2002-2012 by Heiko Oberdiek. ==> 1 page written on pedestrian_orientation_AOS.pdf'.
cyclist_detection_AP : 96.161880 96.238068 91.152649
PDFCROP 1.38, 2012/11/02 - Copyright (c) 2002-2012 by Heiko Oberdiek.
==> 1 page written on cyclist_detection_AP.pdf'. cyclist_orientation_AOS : 95.185928 95.402016 90.347366 PDFCROP 1.38, 2012/11/02 - Copyright (c) 2002-2012 by Heiko Oberdiek. ==> 1 page written on cyclist_orientation_AOS.pdf'.
car_detection_BEV_AP : 86.162407 82.500969 75.850616
PDFCROP 1.38, 2012/11/02 - Copyright (c) 2002-2012 by Heiko Oberdiek.
==> 1 page written on `car_detection_BEV_AP.pdf'.

[ZyHuang97]

I have exactly the same problem.

[KunikaValecha]

I ran into same issue. Well I guess the reason is that testing dataset under kitt/testing actually doesn't hold any labels for groundtruth. If you will read kitti_eval.py carefully then you may find the path to label.txt under eval function. Basically this function was written to perform validation test. But due to some glitches it has been run to do evaluation of testing data which I guess is technically not possible. Kitti doesnt give any groundtruths for testing data(as per my knowledge). So can one evaluate, right?

So basically handling this problem. If you want to see the evaluation i.e. you want to see how good is the confidence percent of your detection then you need to change the path to input data set, which I guess can be manipulated through https://github.com/lzccccc/SMOKE/blob/master/smoke/config/paths_catalog.py this file. However I am not sure, You will have to check it out.

Or if you only want to see the results on your images, then you feed the testing folder with your test data and calib files. Also update test.txt accordingly. And run the code you may see the error persists. But your work will be done in tools folder. There you will find all your results but without the confidence percent. Also to avoid the errors I guess it is safe to comment out do_evaluation function in kitti_eval.py.

[malianghui]

Thanks a lot for making your code open source.

I test kitty dadaset by python tools/plain_train_net.py --eval-only --config-file "configs/smoke_gn_vector.yaml"

And the "smoke_gn_vector.yaml" is:
MODEL:
#WEIGHT: "catalog://ImageNetPretrained/DLA34"
WEIGHT: "/home/mlhui/project/SMOKE/dla34-ba72cf86.pth"
INPUT:
FLIP_PROB_TRAIN: 0.5
SHIFT_SCALE_PROB_TRAIN: 0.3
DATASETS:
DETECT_CLASSES: ("Car", "Cyclist", "Pedestrian")
TRAIN: ("kitti_train",)
#TEST: ("kitti_test",)
TEST: ("kitti_train",)
TRAIN_SPLIT: "trainval"
#TEST_SPLIT: "test"
TEST_SPLIT: "val"
SOLVER:
BASE_LR: 2.5e-4
STEPS: (10000, 18000)
MAX_ITERATION: 25000
IMS_PER_BATCH: 32

But I got the error:

[2020-11-11 19:52:00,830] smoke.engine.inference INFO: Start evaluation on kitti_train dataset(3769 images).
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3769/3769 [02:39<00:00, 23.57it/s]
[2020-11-11 19:54:40,734] smoke.engine.inference INFO: Total run time: 0:02:39.904022 (0.042426113425720655 s / img per device, on 1 devices)
[2020-11-11 19:54:40,749] smoke.engine.inference INFO: Model inference time: 0:02:16.615843 (0.03624723886867897 s / img per device, on 1 devices)
[2020-11-11 19:54:40,749] smoke.data.datasets.evaluation.kitti.kitti_eval INFO: performing kitti detection evaluation:
[2020-11-11 19:54:41,392] smoke.data.datasets.evaluation.kitti.kitti_eval INFO: Evaluate on KITTI dataset
sh: 1: pdfcrop: not found
sh: 1: pdfcrop: not found
sh: 1: pdfcrop: not found
sh: 1: pdfcrop: not found
sh: 1: pdfcrop: not found
sh: 1: pdfcrop: not found
sh: 1: pdfcrop: not found
sh: 1: pdfcrop: not found
sh: 1: pdfcrop: not found
sh: 1: pdfcrop: not found
sh: 1: pdfcrop: not found
sh: 1: pdfcrop: not found
[2020-11-11 19:54:45,985] smoke.data.datasets.evaluation.kitti.kitti_eval INFO: Thank you for participating in our evaluation!
Loading detections...
number of files for evaluation: 3769
done.
save /home/mlhui/project/SMOKE/tools/logs/inference/kitti_train/plot/car_detection.txt
car_detection AP: 0.000000 0.002056 0.002038
save /home/mlhui/project/SMOKE/tools/logs/inference/kitti_train/plot/car_orientation.txt
car_orientation AP: 0.000000 0.000981 0.001078
save /home/mlhui/project/SMOKE/tools/logs/inference/kitti_train/plot/pedestrian_detection.txt
pedestrian_detection AP: 0.000000 0.000000 0.000000
save /home/mlhui/project/SMOKE/tools/logs/inference/kitti_train/plot/pedestrian_orientation.txt
pedestrian_orientation AP: 0.000000 0.000000 0.000000
save /home/mlhui/project/SMOKE/tools/logs/inference/kitti_train/plot/cyclist_detection.txt
cyclist_detection AP: 0.000000 0.000000 0.000000
save /home/mlhui/project/SMOKE/tools/logs/inference/kitti_train/plot/cyclist_orientation.txt
cyclist_orientation AP: 0.000000 0.000000 0.000000
save /home/mlhui/project/SMOKE/tools/logs/inference/kitti_train/plot/car_detection_ground.txt
car_detection_ground AP: 0.000000 0.000000 0.000000
save /home/mlhui/project/SMOKE/tools/logs/inference/kitti_train/plot/pedestrian_detection_ground.txt
pedestrian_detection_ground AP: 0.000000 0.000000 0.000000
save /home/mlhui/project/SMOKE/tools/logs/inference/kitti_train/plot/cyclist_detection_ground.txt
cyclist_detection_ground AP: 0.000000 0.000000 0.000000
save /home/mlhui/project/SMOKE/tools/logs/inference/kitti_train/plot/car_detection_3d.txt
car_detection_3d AP: 0.000000 0.000000 0.000000
save /home/mlhui/project/SMOKE/tools/logs/inference/kitti_train/plot/pedestrian_detection_3d.txt
pedestrian_detection_3d AP: 0.000000 0.000000 0.000000
save /home/mlhui/project/SMOKE/tools/logs/inference/kitti_train/plot/cyclist_detection_3d.txt
cyclist_detection_3d AP: 0.000000 0.000000 0.000000
Your evaluation results are available at:
/home/mlhui/project/SMOKE/tools/logs/inference/kitti_train

can you tell me why? thx!