yukitsuji / 3d_cnn_tensorflow

KITTI data processing and 3D CNN for Vehicle Detection

KiTTI Data Processing and 3D CNN for Vehicle Detection

3D CNN for Vehicle Detection

3D Fully Convolutional Network using PointCloud data for Vehicle Detection
Reference: 3D Fully Convolutional Network for Vehicle Detection in Point Cloud https://arxiv.org/pdf/1611.08069.pdf

Main File is "model_01_deconv.py"
Data Loading Function is "input_velodyne.py"

Example (3D Bounding Box:8 Vertices) Not executing NMS

Requirement

• Python
• tensorflow
• ROS
• Python-PCL (If you will not use pcl, please comment out related code in input_velodyne.py

KiTTI Data Processing

Dataset is KITTI 3D Object Detection Datasets
http://www.cvlibs.net/datasets/kitti/eval_object.php?obj_benchmark=3d

• Velodyne PointCloud
• training labels
• Calibration

3d_cnn_tensorflow's Issues

3D bounding box

Hello, I have hard time to understand why the bounding boxes aren't 3D in my reproduction of the code.
Any idea? thank you

What is the difference between the model_01_deconv and model_025_deconv. Is one used for training over the other. The model_01_deconv training does not seem to work

I tried implementing the training function of the model_01_deconv.py function but I get errors starting with the first being the following:
File "model_01_deconv.py", line 382, in lidar_generator
yield np.array(batch_voxel, dtype=np.float32)[:, :, :, :, np.newaxis], np.array(batch_g_map, dtype=np.float32), np.array(batch_g_cord, dtype=np.float32)
IndexError: too many indices for array

Any specific reasons for choosing these values in the test function - voxel_shape=(360, 400, 40), x=(0, 90), y=(-50, 50), z=(-5.5, 4.5)

The network model question

Hi,
I have studied your code for a period of time. I have a question want to ask you, does the model"velodyne_025_deconv_norm_valid40" you gived was trained by the first network without deconv layer?because i see it's 100x100x5 rather than 200x200x10, I also trained an model with the second network with deconv layer, but when i test i found it is not as effective as the former model, could you share your experinece to me and tell me why? Thanks so much if you can reply please!

Appending temporal information

My main goal is to do a joint detection and tracking, and to achieve this i need to integrate the temporal information in the voxel representation to do the tracking.
Anyone has an idea how should I proceed?
Thank you.

Bounding Box prediction

Can anyone tell me how the bounding boxes are predcition with the model? and which function does that?
Thank you.

Visualize during test time

Hi,

I was wondering if you could tell about how to visualize the result during test time. I know how to use ROS to visualize LIDAR data, but during test when the data starts being outputted to ROS, I am unable to see it in RViz.

Visualizing input_velodyne.py output ?

Hello, I re implemented input_velodyne.py, but I don't know what to expect as output or how to visualize it (because im using it to voxelize the poind cloud data and try to see the result without ROS)
Can anyone be can enough to shed some light on this issue of mine?
Much appreciated!

No such file ‘velodyne/002397.bin’

Hello,
When I was running model_01_deconv.py, the error indicated as follows
No such file or directory: '/home/katou01/download/testing/velodyne/002397.bin'
Could you please tell me what is the problem and how can I solve it?

Best regards

Which dataset should I use?

hi.

the download link in the README.md is a image detection datasets.

• http://www.cvlibs.net/datasets/kitti/eval_object.php

There are 3D and bird-eye view datasets. which one should I use?

One more, in the each datasets pages, there are also multiple files. Do I need all files?

Could you tell me the specific files name or link?

Thanks

filter_car_data(...) function missing

Hi @yukitsuji,

I was trying your code and stumbled upon the function filter_car_data() in model_025_deconv_norm_validate.py.

The implementation for this function seems to be missing.

What is this function for? Can I find it anywhere?

Thanks a lot and best regards,
Mario

def train_test() and def train()

what is the difference between def train_test and def train?
Thank you

support rotate angle detect and classify

hi, from the detect result , it looks like not support detect vehicle with rotate angle, and also didn't support classify, right ? I see other object types(like pedestrian) were filtered except vehicle.
Thanks

Training with new dataset error

Can you please give a brief on how to train this network with new data set.
When i was trying to train it is throwing an error: "ValueError: Tried to convert 'reduction_indices' to a tensor and failed. Error: Argument must be a dense tensor: range(0, 4) - got shape [4], but wanted []."

raw_to_voxel function

I was trying to understand the code but I could wrap my mind around the values of x, y z and resolution (resolution=0.50, x=(0, 90), y=(-50, 50), z=(-4.5, 5.5)).
Can anyone explain this please and what's the point from them? thank you

about

Inference time

For a point cloud in KITTI, how long is the inference time per frame?

TypeError: zip argument #1 must support iteration

Hello, Im was experimenting with the 000058.bin file (where there is only Pedestrain but no cars)
I get the following error on the get_boxcorners() function:
TypeError: zip argument #1 must support iteration
does anyone knows what it means and how to solve it?
Thx

Why do we invert the matrices in the proj_to_velo function in input_velodyne.py

about the environment of the computer

hello, excuse me !
I am a newer of the python, when i run the programs , it always have some errors, maybe i have a wrong with my computer running environment,Could you tell me the operating system and the version of python and tensorflow? I want to learn it ,thank you very much!

Detection and tracking

Hello, i'm trying to develop a model which detects and tracks at the same time, based on this apporach (I'm going to use the Voxel as input for my CNN)
Any idea how should proceed? and can I train my model for detection on the KITTI tracking dataset?
Thank you

The versions of python and tensorflow

Would you like to tell me the versions of python and tensorflow that the code uses? Thanks a lot!

hi, can you tell me how to do the evaluation with 8 pose points?

I use the kitti datasets to train, and can not do its evaluation code by 8 points. So can you tell me how do you do the evaluation?

Deconvolutional layer in model_01_deconv.py

Can anyone explain what the deconvolutional layer def deconv3D_to_output(input_layer, input_dim, output_dim, height, width, length, stride, output_shape, activation=tf.nn.relu, padding="SAME", name="") is for? and does is influence the training/learning process?
Thank you.

The network model question

Hi,
I have studied your code for a period of time. I have a question want to ask you, does the model"velodyne_025_deconv_norm_valid40" you gived was trained by the first network without deconv layer?because i see it's 1001005 rather than 20020010, I also trained an model with the second network with deconv layer, but when i test i found it is not as effective as the former model, could you share your experinece to me and tell me why? Thanks so much if you can reply please!

create_labels() returning empty variables

I tired to execute the this funtion def create_labels(places, size, corners, resolution = 0.5, x=(0, 90), y=(-50, 50), z=(-4.5, 5.5), scale=4, min_value=np.array([0., -50., -4.5])): but the returned variables sphere_center, train_corners have the size of (0,3) and (0,8,3) but they are empty.
Any idea about that?
Thank you

error for running model_01_deconv.py

HI, I have downloaded the object detection data from kitti (velodyne, calib, label (txt files)). when i try to run it I receive this error.
"File "/home/ansab/Downloads/new/3D_CNN_tensorflow-master/model_01_deconv.py", line 384, in lidar_generator
yield np.array(batch_voxel, dtype=np.float32)[:, :, :, :, np.newaxis], np.array(batch_g_map, dtype=np.float32), np.array(batch_g_cord, dtype=np.float32)
IndexError: too many indices for array
Process finished with exit code 134 (interrupted by signal 6: SIGABRT) "

If i give bach size of 1 and give the 1 data file with car in it it runs. But when there is no label with 'car' in label file it gives this error.
Please Help me with this issue my email is [email protected]
Thank you so much

Function filter_car_data(corners) is not defined

Hey Sir!

I am trying to read through your excellent code, but I noticed the function "filter_car_data" called in model_01_deconv.py line: 218 was not defined anywhere. I wonder if there is any file or module where it is defined that I overlooked?

Thank you so much for your help!

Pre trained models

Since my resources are limited, can I use a pre trained model to detect vehicles with this voxelization method?

ImportError: No module named std_msgs.msg

There is a error as the title shows when I run
python model_01_deconv.py

output

3D_CNN_tensorflow/input_velodyne.py", line 10, in <module>
    import std_msgs.msg
ImportError: No module named std_msgs.msg

so, where is the std_msgs?

Function understanding (part 2)

Hello, can anyone explain what is the utility of this function create_objectness_label and what does it do in the train process.
Also, in the train_test() function in model_01_deconv I couldn't find the declation of the cordinate function and What does it do?
And last question, why is there 3 loss function and what is the utility of each one?
Thank you

Cannot See the labels from the xtest data for further evaluation

hi @yukitsuji, this is not an issue I am hoping, but rather a question, please?

I am trying to view the data from the test set to see what data type is challenge for the 3dcnn but unfortunately I didn't see that therein or maybe I missed it.

I have 80 videos of two classes
when I do the split xtrain, ytrain, xtest, ytest,
X_train, Xval_test, Y_train, Yval_test = train_test_split(
X, Y, train_size=0.8, test_size=0.2, random_state=1, stratify=Y, shuffle=True)

Is there a method to visually see the data (not numerically) the actual video or image labels?
for example xtest = vid12, vid24 ,vid11 , vid14, vid29 etc..
its the labels of the video therein xtest I am try to identify.

Why? this would tell me which videos are actually challenging the model and which is not!!
The confusion matrix is stating numerical outputs as well but I still have no way of identifying what videos are challenging the model even if it state 4 misclassifications, there's no way of know those 4 videos in the test. My analysis is blind

Can you guide me with this please?

bird-view projection

I have confirmed code again and again, and couldn't find the projection of object. But there are images which project the object to the bird view.
May I ask how to write the bird-view projection's code?
Thanks a lot!

The network model question

Hi,
I have studied your code for a period of time. I have a question want to ask you, does the model"velodyne_025_deconv_norm_valid40" you gived was trained by the first network without deconv layer?because i see it's 1001005 rather than 20020010, I also trained an model with the second network with deconv layer, but when i test i found it is not as effective as the former model, could you share your experinece to me and tell me why? Thanks so much if you can reply please!

Calibration File

Hey Sir!

Could you please advise me what is the use of camera calibration file in general. I looked into the txt file of it, it is some complicated math matrices involved, so I am having difficulties to relate that to the Velodyne LiDAR input. I wonder if this kind of calibration file necessary for your code, or it is more like a supplementary procedure to increase the accuracy? cause I found that the testing algorithms actually do not utilize anything from the calibration file.

Thank you so much for your help!

Is there a particular code set for converting the output to the KITTI label format?

I am testing this code with the KITTI evaluation development kit which takes in the ground truth labels as well as text files of results. The text files have to be in the same format as the ground truth. The output of the test function in the model_01_deconv script seems to output the coordinates and objectness scores. However, the text file labels need to have the 2D bounding box coordinates with the 3D coordinates of the car. The readme file for the evaluation script shows the structure of the result files:

#Values Name Description

1 type Describes the type of object: 'Car', 'Van', 'Truck',
'Pedestrian', 'Person_sitting', 'Cyclist', 'Tram',
'Misc' or 'DontCare'
1 truncated Float from 0 (non-truncated) to 1 (truncated), where
truncated refers to the object leaving image boundaries
1 occluded Integer (0,1,2,3) indicating occlusion state:
0 = fully visible, 1 = partly occluded
2 = largely occluded, 3 = unknown
1 alpha Observation angle of object, ranging [-pi..pi]
4 bbox 2D bounding box of object in the image (0-based index):
contains left, top, right, bottom pixel coordinates
3 dimensions 3D object dimensions: height, width, length (in meters)
3 location 3D object location x,y,z in camera coordinates (in meters)
1 rotation_y Rotation ry around Y-axis in camera coordinates [-pi..pi]
1 score Only for results: Float, indicating confidence in
detection, needed for p/r curves, higher is better.

Thanks in advance!

Voxelize multiple frames at once

Can anyone tell me how can I voxelize a batch of frames at once instead of voxelizing a frame at a time? thank you

How much memory needed for training?

I used tensorflow-CPU for training dataset with 48G memory. But I get memory fault after training for several hours :
terminate called after throwing an instance of 'std::bad_alloc'
what(): std::bad_alloc

What is the parameter “scale” used for？

Hi, what is the parameter “scale” used for？

How to run the program

Hi,
Would you like to share how to run the program? Usually in ROS, they begin by using ".launch" file, but I don't find it on your project.

Besides, I'm using anaconda environment and I can't install the rospy package on the environment, can you tell me how to install it on anaconda environment?

Is there any specific reason for not using Deconv layer for objectness map and bounding box map?

hi @yukitsuji Paper mentions to use deconv for output layer to upsample the objectness and boundingbox.
Could you comment on that?
Thanks

Functions Understanding

Hello,
Im trying to re implement the volxelization process as I need it later on. But I'm having hard time understanding some function as I'm new to Python.
I hope someone could be kind enough to walk me through them and explain what do they do exactly. These are the function:

---

def filter_camera_angle(places):
def create_publish_obj(obj, places, rotates, size):
def publish_pc2(pc, obj):
center_to_sphere(places, size, resolution=0.50, min_value=np.array([0., -50., -4.5]), scale=4, x=(0, 90), y=(-50, 50), z=(-4.5, 5.5)):
def sphere_to_center(p_sphere, resolution=0.5, scale=4, min_value=np.array([0., -50., -4.5])):
def voxel_to_corner(corner_vox, resolution, center):#TODO
def corner_to_train(corners, sphere_center, resolution=0.50, x=(0, 90), y=(-50, 50), z=(-4.5, 5.5), scale=4, min_value=np.array([0., -50., -4.5])):
def corner_to_voxel(voxel_shape, corners, sphere_center, scale=4):

---

Much appreciated!