ndrplz / surround_vehicles_awareness

Learn to map surrounding vehicles onto a bird's eye view of the scene.

Learning to Map Vehicles into Bird's Eye View

This code accompanies the paper "Learning to map surrounding vehicles into bird's eye view using synthetic data".

It contains the code for loading data and pre-trained SDPN model proposed in the paper.

How-to-run

Script entry-point is in main.py.

When main.py is run, pretrained weights are automatically downloaded and injected in the model.

Model is then used to perform and inference on a sample data, mapping a car from the dashboard camera view to the bird's eye view of the scene. If everything works correctly, the output should look like this.

Dependencies

The code was developed with the following configuration:

• python 2.7.11
• numpy 1.11.2
• opencv 3.1.0
• Theano 0.9.0.dev3
• Keras 1.1.2

Other configuration will reasonably work, but have never been explicitly tested.

Dataset

In this repository only one example is provided, to the end of verifying that the model is working correctly.

The whole dataset, which comprises more than 1M couples of bounding boxes, can be found here.

To get an idea of how the data look like you can check this video.

surround_vehicles_awareness's Issues

Could you please provide the camera intrinsic and extrinsic parameters?

Thanks for providing the dataset, I've already download the images.

Could you please provide the camera intrinsic and extrinsic parameters or the Homography matrix?
If so, it's really helpful for my research.
Thank you very much!

Annotation tool

Hi,
What the annotation tool do you use it ? I try "labelimg", it is not fit

Do you know any GTA based Synthetic dataset for vehicle ReID?

Hi, I am currently working on vehicle ReID, your project which aims to map front view to bird view seems to be a particular type of ReID. Did you ever generate a vehicle ReID dataset or happens to know one? Thanks!

Couldn't download pre-trained weights

When I download http://imagelab.ing.unimore.it/files/pretrained_models/keras/SPDN_w.hdf5," HTTP Error 404.3 - Not Found" occurs.

Bird-eye view data generation

Thanks for this amazing project, and keep up the good work.

Regarding the bird-eye view data generation, is it possible if you could share with me how this was achieved using deep GTAV and script hook v because I can only generate the frontal view images with annotations ?

Model was constructed with shape (None, None, None, 3) for input KerasTensor, but it was called on an input with incompatible shape (None, 3, 224, 224)

When I am running the project I am getting the following error:
Model was constructed with shape (None, None, None, 3) for input KerasTensor, but it was called on an input with incompatible shape (None, 3, 224, 224)

Could you please tell me how to fix this issue?

main.py not runnning

I clone the git repository and tried running the project but following error is coming:
Traceback (most recent call last):
File "main.py", line 14, in
model = SDPN(summary=True)
File "/data/priyam/surround_vehicles_awareness/model.py", line 30, in SDPN
crop_encoded = resnet(input_crop) # shape of crop_encoded is 2018x1x1
File "/usr/local/lib/python2.7/dist-packages/Keras-1.1.1-py2.7.egg/keras/engine/topology.py", line 517, in call
self.add_inbound_node(inbound_layers, node_indices, tensor_indices)
File "/usr/local/lib/python2.7/dist-packages/Keras-1.1.1-py2.7.egg/keras/engine/topology.py", line 571, in add_inbound_node
Node.create_node(self, inbound_layers, node_indices, tensor_indices)
File "/usr/local/lib/python2.7/dist-packages/Keras-1.1.1-py2.7.egg/keras/engine/topology.py", line 155, in create_node
output_tensors = to_list(outbound_layer.call(input_tensors[0], mask=input_masks[0]))
File "/usr/local/lib/python2.7/dist-packages/Keras-1.1.1-py2.7.egg/keras/engine/topology.py", line 2075, in call
output_tensors, output_masks, output_shapes = self.run_internal_graph(inputs, masks)
File "/usr/local/lib/python2.7/dist-packages/Keras-1.1.1-py2.7.egg/keras/engine/topology.py", line 2217, in run_internal_graph
output_tensors = to_list(layer.call(computed_tensor, computed_mask))
File "/usr/local/lib/python2.7/dist-packages/Keras-1.1.1-py2.7.egg/keras/layers/convolutional.py", line 466, in call
filter_shape=self.W_shape)
File "/usr/local/lib/python2.7/dist-packages/Keras-1.1.1-py2.7.egg/keras/backend/tensorflow_backend.py", line 1683, in conv2d
x = tf.nn.conv2d(x, kernel, strides, padding=padding)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/gen_nn_ops.py", line 394, in conv2d
data_format=data_format, name=name)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/op_def_library.py", line 749, in apply_op
op_def=op_def)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 2382, in create_op
set_shapes_for_outputs(ret)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 1783, in set_shapes_for_outputs
shapes = shape_func(op)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/common_shapes.py", line 596, in call_cpp_shape_fn
raise ValueError(err.message)
ValueError: Dimensions must be equal, but are 224 and 3

ValueError: number of input channels does not match corresponding dimension of filter

when running the main.py to build the model based on Resnet50, the following code is shown below

input_crop = Input(shape=(3, 224, 224))

extract feature from image crop

resnet = ResNet50(include_top=False, weights='imagenet')
for layer in resnet.layers: # set resnet as non-trainable
layer.trainable = False

crop_encoded = resnet(input_crop)
However, I got an error

'ValueError: number of input channels does not match corresponding dimension of filter, 224 != 3'
how can I fix it?

Datafile column labels

Hi thanks for making this project and dataset available.

I was wondering if you had the names of the columns for the unfiltered bboxes.txt files. It looks like there's some more information in the file that could be useful and I'd like to use it if possible.

Thanks!

filtered dataset

Hi thank you for sharing your dataset, it's amazing! But i was wondering
if I wanted to collect data other than the distance from of the BBox, how can i then filter the dataset like you did? I'm trying to build a model to predict the speed of vehicles. Thank you!

a small dataset provided

@ndrplz Thanks for the code and dataset, however the dataset is way too big for me to download, would it be ok to have a smaller dataset to train ?

Unable to get SVA dataset

Hi,

The aimage lab dataset link seems broken.Is there anyway to get the GTA V SVA dataset ?

Regards

any clue? versions are not exactly the same,

/local/lib/python2.7/site-packages/keras_applications/resnet50.py:265: UserWarning: The output shape of ResNet50(include_top=False) has been changed since Keras 2.2.0.
warnings.warn('The output shape of ResNet50(include_top=False) '
Traceback (most recent call last):
File "main.py", line 14, in
model = SDPN(summary=True)
File "/home/nvidia/surround_vehicles_awareness/model.py", line 29, in SDPN
crop_encoded = resnet(input_crop) # shape of crop_encoded is 2018x1x1
File "/home/nvidia/env2/local/lib/python2.7/site-packages/keras/engine/base_layer.py", line 457, in call
output = self.call(inputs, **kwargs)
File "/home/nvidia/env2/local/lib/python2.7/site-packages/keras/engine/network.py", line 564, in call
output_tensors, _, _ = self.run_internal_graph(inputs, masks)
File "/home/nvidia/env2/local/lib/python2.7/site-packages/keras/engine/network.py", line 721, in run_internal_graph
layer.call(computed_tensor, **kwargs))
File "/home/nvidia/env2/local/lib/python2.7/site-packages/keras/layers/convolutional.py", line 171, in call
dilation_rate=self.dilation_rate)
File "/home/nvidia/env2/local/lib/python2.7/site-packages/keras/backend/tensorflow_backend.py", line 3650, in conv2d
data_format=tf_data_format)
File "/home/nvidia/env2/local/lib/python2.7/site-packages/tensorflow/python/ops/nn_ops.py", line 850, in convolution
data_format=data_format)
File "/home/nvidia/env2/local/lib/python2.7/site-packages/tensorflow/python/ops/nn_ops.py", line 937, in init
filter_shape[num_spatial_dims]))
ValueError: number of input channels does not match corresponding dimension of filter, 224 != 3

ValueError: total size of new array must be unchanged

Ubuntu 16.04

home/danfouer/.conda/envs/keras/bin/python /home/danfouer/Documents/pycharm/Project/SVA/main.py
Using Theano backend.
Traceback (most recent call last):
File "/home/danfouer/Documents/pycharm/Project/SVA/main.py", line 16, in
model = SDPN(summary=True)
File "/home/danfouer/Documents/pycharm/Project/SVA/model.py", line 30, in SDPN
crop_encoded = Reshape(target_shape=(2048,))(crop_encoded)
File "/home/danfouer/.conda/envs/keras/lib/python2.7/site-packages/keras/engine/base_layer.py", line 480, in call
output_shape = self.compute_output_shape(input_shape)
File "/home/danfouer/.conda/envs/keras/lib/python2.7/site-packages/keras/layers/core.py", line 404, in compute_output_shape
input_shape[1:], self.target_shape)
File "/home/danfouer/.conda/envs/keras/lib/python2.7/site-packages/keras/layers/core.py", line 392, in _fix_unknown_dimension
raise ValueError(msg)
ValueError: total size of new array must be unchanged

Error using theano

Hi,

Thank you for sharing your code. I am have issues about your code.
input_crop = Input(shape=(3, 224, 224)) changed to input_crop = Input(shape=(224, 224, 3))
it can run but there is error like this
Traceback (most recent call last):
File "main.py", line 28, in
Y_pred = model.predict([X_coords, X_crops])
File "C:\Users\user\anaconda3\envs\py2\lib\site-packages\keras\engine\training.py", line 1166, in predict
check_batch_dim=False)
File "C:\Users\user\anaconda3\envs\py2\lib\site-packages\keras\engine\training.py", line 111, in standardize_input_data
str(array.shape))
Exception: Error when checking : expected input_2 to have shape (None, 224, 224, 3) but got array with shape (1L, 3L, 224L, 224L)

Could you help me for solve this issues ? Thank you