maudzung / sfa3d

Super Fast and Accurate 3D Object Detection based on 3D LiDAR Point Clouds (The PyTorch implementation)

Welcome to Nguyen Mau Dzung's GitHub 👋

sfa3d's Issues

Have you compared SFA3D with complex_yolov4? Which algorithm is better?

loss reduce so slow

tensor(1.9749, device='cuda:0')
tensor(1.8765, device='cuda:0')
tensor(1.9162, device='cuda:0')
tensor(1.6088, device='cuda:0')
tensor(1.7206, device='cuda:0')
tensor(2.1326, device='cuda:0')
tensor(1.7767, device='cuda:0')
tensor(2.1650, device='cuda:0')
tensor(1.9944, device='cuda:0')
epoch: 336======================================
save a checkpoint at ./checkpoint/fpn_resnet_34/Model_fpn_resnet_34_epoch_336.pth
tensor(2.5376, device='cuda:0')
tensor(1.9334, device='cuda:0')
tensor(1.6190, device='cuda:0')
tensor(2.1241, device='cuda:0')
tensor(2.0782, device='cuda:0')
tensor(1.8621, device='cuda:0')
tensor(1.7853, device='cuda:0')
tensor(1.7603, device='cuda:0')
tensor(1.9144, device='cuda:0')
epoch: 337======================================
save a checkpoint at ./checkpoint/fpn_resnet_34/Model_fpn_resnet_34_epoch_337.pth
tensor(1.6354, device='cuda:0')
tensor(1.8733, device='cuda:0')
tensor(1.6733, device='cuda:0')
tensor(1.9550, device='cuda:0')
tensor(1.7354, device='cuda:0')
tensor(2.1533, device='cuda:0')
tensor(1.8266, device='cuda:0')
tensor(1.7848, device='cuda:0')
tensor(2.2798, device='cuda:0')
epoch: 338======================================
save a checkpoint at ./checkpoint/fpn_resnet_34/Model_fpn_resnet_34_epoch_338.pth
tensor(1.7751, device='cuda:0')
tensor(1.8832, device='cuda:0')
tensor(1.6201, device='cuda:0')
tensor(2.1500, device='cuda:0')

loss reducing so slow, and evaluation not good

How to run it on Kitti odometry dataset?

I'm sorry. I'm a fresh man to deep learning. Now, I can run the demo_front.py to run the kitti detection test data. But how to use it in kitti odometry velodyne dataset ? such as 00 sequence ? Can you give me some examples？

cv2.line(img, (corners_int[0, 0], corners_int[0, 1]), (corners_int[3, 0], corners_int[3, 1]), (255, 255, 0), 2)

Solution :

cv2.line(img, (int(corners_int[0, 0]), int(corners_int[0, 1])), (int(corners_int[3, 0]), int(corners_int[3, 1])), (255, 255, 0), 2)

Looking forward to the ROS documentation

AttributeError: module 'rospkg' has no attribute 'RosPack'

can someone provide pre-trained weights on waymo please ?

Hello, If someone can provide pre-trained weights on data waymo, it would be very helpful !

Evaluation

Hi, Nguyen. I evaluated this work by using the 'kitti eval' tool. As far as I know, this SFA3D has the same detection range as your work of https://github.com/maudzung/Complex-YOLOv4-Pytorch and https://github.com/maudzung/YOLO3D-YOLOv4-PyTorch/. However, this work has a better performance than the other two work. Please tell me if my conclusion about these three methods' performance is right?

i have found resnet50 has better performance than fpn_reset18/34

cause info from pointcloud has not space structure, so it's results not so good, anchor free's method maybe not useful

Problem with training on Waymo

Dear @maudzung,

first, thank you for your work.
@lucabart97 and I, have spent the last week trying to train this network on the Waymo dataset.
We first used your other repo to convert Waymo to Kitti format.
Then we changed the train.txt and val.txt files and also kitti_config.txt.
However, we were not able to achieve any good results with the training.
Indeed, the train loss has a very weird behavior:

Can you give us some tips to make it work? Or point out what we are doing wrong?

Thank you very much in advance.

Best,
Micaela

What part can I refer to to detect continuously?

The object disappears from the front and rear of the vehicle through the image and is detected again. What part can I refer to to detect continuously? Is the point cloud divided into the front and the rear to be detected? Thank you for providing great work.

How to train my own dataset?

Thanks for your great work!
Is there any idea for training my own dataset to test this project?

RuntimeError while training FPN-Resnet18/34/50

I got this error while trying to train fpn-resnet18/34/50 model on kitti.
It seems that out_layer3 outputs torch.Size([16, 256, 39, 39]) while up_level1 outputs torch.Size([16, 512, 40, 40]). Please can you suggest how to get rid of this error?

Traceback (most recent call last):
  File "train.py", line 279, in <module>
    main()
  File "train.py", line 67, in main
    main_worker(configs.gpu_idx, configs)
  File "train.py", line 161, in main_worker
    train_one_epoch(train_dataloader, model, optimizer, lr_scheduler, epoch, configs, logger, tb_writer)
  File "train.py", line 211, in train_one_epoch
    outputs = model(imgs)
  File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/docker_sfa3d/SFA3D/sfa/models/fpn_resnet.py", line 181, in forward
    concat_level1 = torch.cat((up_level1, out_layer3), dim=1)
RuntimeError: Sizes of tensors must match except in dimension 2. Got 39 and 40 (The offending index is 0)

Reference and performance

Hello, thank you for your great work. I’m an intern who work on perception 3D for autonomous vehicles and I would like to use your project Super-Fast-Accurate-3D-Object-Detection-PyTorch. I have a suggestion : Is there a specific reference on this method and also is there comparison ((AP metric and inference time) with others methods on dataset like KITTI ?

Thank you.

New train model can not be used

ROS can not find pretrained weights

Problem:
Running the Nodes causes it to stop immediately stating that it can not find the file fpn_resnet_18_epoch_300.pth

Solution:

In rosInference.py L119 it tries to load the weights:

package_path = rospack.get_path('super_fast_object_detection')
.....
configs.pretrained_path = package_path + '/checkpoints/fpn_resnet_18/fpn_resnet_18_epoch_300.pth'

But those are not contained in package_path which is /some_folders/some_more_folders/SFA3D/ros/srcsuper_fast_object_detection/. The weights are in the folder /checkpoints in the main root of this git.

A not workaround would be:

configs.pretrained_path = package_path + '/../../../checkpoints/fpn_resnet_18/fpn_resnet_18_epoch_300.pth'

Otherwise one could rethinking the whole structure. Maybe another ros repo should be opend where this repo is included as a submodule. This would also have some benefits like version control: The ros maintainer can state the to be used commit from the SFA3D main repo. If SFA3D maintainer updates SFA3D - maybe with with breaking changes - it would not break the ros node.

How long it will take to reimplement this code in tensorflow?

Hey maudzung,

I am thinking to re implement your work in tensorflow. Could you please explain how long it will take if I start now?

I have limited time to do so. that is the reason.

Thanks in advance.

can i run it on windows 10 machine with cpu only (no gpu)?

ROS

I get the following error when I run rosrun super_fast_object_detection rosInference.py

[ERROR] [1622813649.547097]: bad callback: <function on_scan at 0x7fafb9fdef70>
Traceback (most recent call last):
File "/opt/ros/noetic/lib/python3/dist-packages/rospy/topics.py", line 750, in _invoke_callback
cb(msg)
File "/home/ahmed/PCL_OD/SFA3D/ros/src/super_fast_object_detection/src/rosInference.py", line 63, in on_scan
bev_map = makeBEVMap(front_lidar, cnf.boundary)
File "/home/ahmed/PCL_OD/SFA3D/sfa/data_process/kitti_bev_utils.py", line 34, in makeBEVMap
_, unique_indices, unique_counts = np.unique(PointCloud[:, 0:2], axis=0, return_index=True, return_counts=True)
File "<array_function internals>", line 5, in unique
File "/home/ahmed/anaconda3/lib/python3.8/site-packages/numpy/lib/arraysetops.py", line 275, in unique
ar = ar.reshape(orig_shape[0], -1)
ValueError: cannot reshape array of size 0 into shape (0,newaxis)

Any tips to resolve the error?

measuring the distance

do you have any idea how to measuring the distance of detected objects? I mean instate of presenting the FPS I want to display the distance of an object?

overfitting

I'm really appreciate you for the nice source code. With my own datasets, i got the nice loss of training , but in validation loss increase , i think may be the model is overfitting . Can you recommand me some way to improve the accuracy ?
lr=0.0006, epochs=120 ,6630 frames for training

Front view

thank you for this work, can you please provide the same method with a Front view solution not BEV??

Run on single GPU

hi,
how can I run this code on a single GPU?

Performance and speed

Hello, maudzuang!
Thanks for sharing your nice work SFA3D! Can you provide its detection and speed performance on KITTI 3D Object Detection Dataset in README.md.

Bad Loss Of Model

Hi author,
I'm really appreciate you for the nice source code. But when i set up enviroment for training , i can't achieve good result like your repo . I got the nice loss of training , but in validation loss increase , i think may be the model is overfitting . May be can you recommand me some way to improve the accuracy ?
Thanks !

IndexError

When I run visualization, inference, and demo scripts, there always occurs this error for some specific frame:

Original Traceback (most recent call last):
  File "/home/zwq/miniconda2/envs/pytorch/lib/python3.7/site-packages/torch/utils/data/_utils/worker.py", line 178, in _worker_loop
    data = fetcher.fetch(index)
  File "/home/zwq/miniconda2/envs/pytorch/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/home/zwq/miniconda2/envs/pytorch/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 44, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/home/zwq/workspace/SFA3D/sfa/data_process/kitti_dataset.py", line 66, in __getitem__
    return self.load_img_only(index)
  File "/home/zwq/workspace/SFA3D/sfa/data_process/kitti_dataset.py", line 76, in load_img_only
    bev_map = makeBEVMap(lidarData, cnf.boundary)
  File "/home/zwq/workspace/SFA3D/sfa/data_process/kitti_bev_utils.py", line 37, in makeBEVMap
    IHD_Map[0, np.int_(PointCloud_top[:, 0]), np.int_(PointCloud_top[:, 1])] = PointCloud_top[:, 3]
IndexError: index 608 is out of bounds for axis 2 with size 608

How can I fix it?

High Positive False Issue

Dear @maudzung

Thanks for your great work! We have trained SFA3D in the Waymo dataset. It seems great but still a problem. The False positive is common. Like in this picture, the bounding box is shown in a place without a car even when I set the thresh as 0.4. May you give us some tips to improve the result? Thanks

Zhimin Chen

The Question about single GPU to test the model（run demo_front.py）

Traceback (most recent call last):
File "demo_front.py", line 47, in
model.load_state_dict(torch.load(configs.pretrained_path))
File "/home/robotics/Application/Anaconda/anaconda3/envs/Object_Detection/lib/python3.6/site-packages/torch/serialization.py", line 593, in load
return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)
File "/home/robotics/Application/Anaconda/anaconda3/envs/Object_Detection/lib/python3.6/site-packages/torch/serialization.py", line 773, in _legacy_load
result = unpickler.load()
File "/home/robotics/Application/Anaconda/anaconda3/envs/Object_Detection/lib/python3.6/site-packages/torch/serialization.py", line 729, in persistent_load
deserialized_objects[root_key] = restore_location(obj, location)
File "/home/robotics/Application/Anaconda/anaconda3/envs/Object_Detection/lib/python3.6/site-packages/torch/serialization.py", line 178, in default_restore_location
result = fn(storage, location)
File "/home/robotics/Application/Anaconda/anaconda3/envs/Object_Detection/lib/python3.6/site-packages/torch/serialization.py", line 154, in _cuda_deserialize
device = validate_cuda_device(location)
File "/home/robotics/Application/Anaconda/anaconda3/envs/Object_Detection/lib/python3.6/site-packages/torch/serialization.py", line 148, in validate_cuda_device
device=device, device_count=torch.cuda.device_count()))
RuntimeError: Attempting to deserialize object on CUDA device 1 but torch.cuda.device_count() is 1. Please use torch.load with map_location to map your storages to an existing device.

My device only have one GPU GTX1660 Ti, So, where should I modify the code to successful run the demo_front.py？

Did you ever try the resnet50, resnet101 and resnet 152 in your work?

False Object detection when running on autoware ROSbags

Used wget https://autoware-ai.s3.us-east-2.amazonaws.com/sample_moriyama_150324.tar.gz ROSbag file,
but many times false objects were detected.

Could you please share a sample data-set to validate the ROS inference node?

How you are doing data augmentation?

Hi Maudzung,

Thank you for your great work. But I just need to know how you are doing data augmentation and where?
Could you please give me the hyperlink to data augmentation in your implementation?

I tried to find data augmentation, but I could not make it.

Thanks in advance

NameError: name 'labels' is not defined

File "training.py", line 502, in <module> train() File "training.py", line 197, in train main_worker(configs.gpu_idx, configs) File "training.py", line 313, in main_worker actual_detections_2d = get_2d_mapping(labels, type='label') NameError: name 'labels' is not defined

after validation check it giving me this error

run model live

How can we use the model on a lidar sensor? can you add some explanation for that?

Visualization of predictions compared with Ground Truth

Great work! Could you pls provide how to do visualization of predictions compared with Ground Truth?

Evaluation report in README

Hi,
Could you add evaluation report (with average FPS and detection accuracy) to README?
Thank you.

Add the pointpillar feature network instead of the human made picture

Hello, have you ever tried to add the pointpillar feature network instead of the human made picture of pointcloud? And how to choose the number of the batchsize and the epoch when you training this work? I would appreciate that if you can reply on me.

Error in visualizing dataset

Hello..

I have got the below message in visualizing dataset step. What should I do ?

ubuntu@n:~/Documents/SFA3D/sfa/data_process$ python3 kitti_dataset.py
Press n to see the next sample >>> Press Esc to quit...
Traceback (most recent call last):
File "kitti_dataset.py", line 296, in
bev_map, labels, img_rgb, img_path = dataset.draw_img_with_label(idx)
File "kitti_dataset.py", line 254, in draw_img_with_label
img_path, img_rgb = self.get_image(sample_id)
File "kitti_dataset.py", line 116, in get_image
img = cv2.cvtColor(cv2.imread(img_path), cv2.COLOR_BGR2RGB)
cv2.error: OpenCV(4.2.0) /io/opencv/modules/imgproc/src/color.cpp:182: error: (-215:Assertion failed) !_src.empty() in function 'cvtColor'

Cuda out of memory

hello, i am currently always facing a cuda out of memory error when i run the train.py file, i looked though the internet it said that i needed to lower the batch size but i don't know how to dot that, there isn't like a clear batch size number or value i can decrease, so can you please help me.

ROS node setup error

I believe that python 2.7 is standard in ROS melodic, but pip install asks for pip3.
Please let me know if you have any ideas.

Environment

• ubuntu 18.04
• ros melodic

~/workspace/catkin_ws/src/SFA3D master
❯ pip install .
DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
Defaulting to user installation because normal site-packages is not writeable
Processing /home/yukihiro/workspace/catkin_ws/src/SFA3D
ERROR: Package 'sfa' requires a different Python: 2.7.17 not in '>=3.6.0'

Using the model trained on the kitti dataset, testing our own dataset

hi，guys，I used the kitti dataset from the project for training and the resulting model was used to detect the vehicles in my own dataset and why it was invalid.My own dataset has been converted to kitti format (.bin) and is displayed correctly in the test, but it does not show the red box in the point cloud plot.

bev not good, suggest one

cpoint_cloud = np.copy(point_cloud)

rgb_map = np.zeros((3, cnf.BEV_HEIGHT, cnf.BEV_WIDTH), dtype=np.float32)
rgb_map_cnt = np.zeros((cnf.BEV_HEIGHT, cnf.BEV_WIDTH), dtype=np.int)
max_height = float(cnf.boundary['maxZ'] - cnf.boundary['minZ'])
for point in cpoint_cloud:
	x = np.floor(point[0] / cnf.DISCRETIZATION_X) + np.int_(cnf.OFFSET_X)
	y = np.floor(point[1] / cnf.DISCRETIZATION_Y) + np.int_(cnf.OFFSET_Y)
	x = int(x)
	y = int(y)
	if x >= cnf.BEV_HEIGHT or y >= cnf.BEV_WIDTH:
		continue
	rgb_map[0, x, y] += point[3] / 255.0
	rgb_map[1, x, y] += (point[2] - cnf.boundary['minZ']) / max_height
	rgb_map_cnt[x, y] += 1

for point in cpoint_cloud:
	x = np.floor(point[0] / cnf.DISCRETIZATION_X) + np.int_(cnf.OFFSET_X)
	y = np.floor(point[1] / cnf.DISCRETIZATION_Y) + np.int_(cnf.OFFSET_Y)
	x = int(x)
	y = int(y)
	if x >= cnf.BEV_HEIGHT or y >= cnf.BEV_WIDTH:
		continue
	if 0 == rgb_map[2, x, y]:
		rgb_map[0, x, y] /= rgb_map_cnt[x, y]
		rgb_map[1, x, y] /= rgb_map_cnt[x, y]
		rgb_map[2, x, y] = np.minimum(1.0, np.log(rgb_map_cnt[x, y]+1) / np.log(cnf.unique_max))

Làm thế nào để tạo Custom dataset?

Hi Dũng,
Mình muốn train với tập dữ liệu tự tạo
Làm thế nào để tạo các file dữ liệu bên dưới nhỉ:
Velodyne point clouds
Training labels of object data set
Camera calibration matrices of object data set
Left color images of object data set (12 GB)

Cám ơn

On running the pretrained model, an error occurs

using ResNet architecture with feature pyramid

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=

Traceback (most recent call last):
File "test.py", line 113, in
model.load_state_dict(torch.load(configs.pretrained_path))
File "C:\Users\Asus\AppData\Local\Programs\Python\Python36\lib\site-packages\torch\serialization.py", line 593, in load
return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)
File "C:\Users\Asus\AppData\Local\Programs\Python\Python36\lib\site-packages\torch\serialization.py", line 773, in _legacy_load
result = unpickler.load()
File "C:\Users\Asus\AppData\Local\Programs\Python\Python36\lib\site-packages\torch\serialization.py", line 729, in persistent_load
deserialized_objects[root_key] = restore_location(obj, location)
File "C:\Users\Asus\AppData\Local\Programs\Python\Python36\lib\site-packages\torch\serialization.py", line 178, in default_restore_location
result = fn(storage, location)
File "C:\Users\Asus\AppData\Local\Programs\Python\Python36\lib\site-packages\torch\serialization.py", line 154, in _cuda_deserialize
device = validate_cuda_device(location)
File "C:\Users\Asus\AppData\Local\Programs\Python\Python36\lib\site-packages\torch\serialization.py", line 148, in validate_cuda_device
device=device, device_count=torch.cuda.device_count()))
RuntimeError: Attempting to deserialize object on CUDA device 1 but torch.cuda.device_count() is 1. Please use torch.load with map_location to map your storages to an existing device.

I see the issue is because of the GPU count being 1, could anybody help me with how to do it on a machine with a single GPU?

KITTI:Overfitting

I'm really appreciate you for the nice source code，I download your code and train with the Kitti dataset， the val loss curve is as follows:

When I use the fpn_resnet_18_epoch_300.pth to pretrain, the val loss curve is as follows:

Can you give us some tips to make it work?

Using my own data

Hello, I want to create data using 3 Lidars and use your program. There are bin files and png files that I made.

But, how can I create a txt file to go into the Calib folder?

There are other github packages that calibrate Lidar and Camera.
However, I cannot fully understand the details of 000000.txt,
and I cannot calibrate because I cannot create my own txt file that plays the exact same role as the txt file in the Calib folder.

1. If you know the contents of 000000.txt, can you tell me?
2. Can I create a txt file with my camera and Lidars?

KeyError: 'optimizer' when resuming training from checkpoint

I am trying to resume training from the checkpoint provided in the repo (fpn_resnet_18_epoch_300.pth), and when I run python train.py --gpu_idx 0 --resume_path ../checkpoints/fpn_resnet_18/fpn_resnet_18_epoch_300.pth I get this error:

Traceback (most recent call last): File "train.py", line 281, in <module> main() File "train.py", line 67, in main main_worker(configs.gpu_idx, configs) File "train.py", line 132, in main_worker optimizer.load_state_dict(utils_state_dict['optimizer']) KeyError: 'optimizer'

How can I use a Lidar for live feed to the network?

I want to use just a Lidar like Ydlidar to get live detection from the point cloud.
Can You guide me?

Can you provide a script evaluate.py please ?

Hello, thanks you for your great work! It's very interesting. I would like to evaluate this model (AP per classe and mAP) like ComplexYolov4. Can you provide a script evaluate.py please?

thank you.

how to get boundry parameters?

in kitti_config.py, there is paramter boundary and the values is boundary = {
"minX": 0,
"maxX": 50,
"minY": -25,
"maxY": 25,
"minZ": -2.73,
"maxZ": 1.27
}.

so my question is how i get this paramter values from my dataset. i have .las and .bin point cloud files only. also i dont have calib files. so can i used kitti calib files?

Usage in ROS

There is little information of the usage in ROS.
Could you update the README and give more information?
By the way, I want to use this to deal with the point cloud from a camera(like stereo camera), is it OK?