(ECCV 2020) PiP: Planning-informed Trajectory Prediction for Autonomous Driving

Home Page: http://song-haoran.com/planning-informed-prediction/

Python 65.08% MATLAB 34.44% Shell 0.48%

deep-learning motion-forecasting trajectory-prediction trajectory-planning autonomous-driving eccv2020

pip-planning-informed-prediction's Introduction

Planning-informed Trajectory Prediction (PiP)

The official implementation of "PiP: Planning-informed Trajectory Prediction for Autonomous Driving" (ECCV 2020),

by Haoran Song, Wenchao Ding, Yuxuan Chen, Shaojie Shen, Michael Yu Wang and Qifeng Chen.

Inform the multi-agent future prediction with ego vehicle's planning in a novel planning-prediction-coupled pipeline.

For more details, please refer to our project website / paper / arxiv.

Dependencies

conda create -n PIPrediction python=3.7
source activate PIPrediction

conda install pytorch==1.2.0 torchvision==0.4.0 cudatoolkit=10.0 -c pytorch
conda install tensorboard=1.14.0
conda install numpy=1.16 scipy=1.4 h5py=2.10 future

Download

Raw datasets: download NGSIM and highD, then process them into the required format (.mat) using the preprocessing code.
Processed datasets: download from this link and save them in datasets/.
Trained models: download from this link and save them in trained_models/.

Running

Training by sh scripts/train.sh or running

python train.py --name ngsim_demo --batch_size 64 --pretrain_epochs 5 --train_epochs 10 \
    --train_set ./datasets/NGSIM/train.mat \
    --val_set ./datasets/NGSIM/val.mat

Test by sh scripts/test.sh or running

python evaluate.py --name ngsim_model --batch_size 64 \
    --test_set ./datasets/NGSIM/test.mat

Documentation

model.py : It contains the concrete details of the proposed PiP architecture.
train.py : It contains the detailed approach for training PiP model. All the network parameters are provided by the default values.
evaluate.py : It contains the approach for evaluating a trained model. The prediction precision is reported by RMSE & NLL values at future time frames.
data.py : It contains the customized dataset class for handling and batching trajectory data
utils.py : It contains the loss calculation functions and some other helper functions.
preprocess/ : It contains Matlab code for preprocessing the raw data from NGSIM or HighD into the required format.

Citation

If you find our work useful in your research, please citing:

@InProceedings{song2020pip,
author = {Song, Haoran and Ding, Wenchao and Chen, Yuxuan and Shen, Shaojie and Wang, Michael Yu and Chen, Qifeng},
title = {PiP: Planning-informed Trajectory Prediction for Autonomous Driving},
booktitle = {The European Conference on Computer Vision (ECCV)},
month = {August},
year = {2020}
}

pip-planning-informed-prediction's People

Contributors

Stargazers

Watchers

pip-planning-informed-prediction's Issues

Looking forward to open source

Very good work! Looking forward to your open source, I hope to learn something!

报错问题

您好，我有一个问题，当我关闭planning模块时，evaluate会报错：RuntimeError: Error(s) in loading state_dict for pipNet:
Unexpected key(s) in state_dict: "plan_lstm.weight_ih_l0", "plan_lstm.weight_hh_l0", "plan_lstm.bias_ih_l0", "plan_lstm.bias_hh_l0", "plan_conv_social.0.weight", "plan_conv_soc
ial.0.bias", "plan_conv_social.3.weight", "plan_conv_social.3.bias".
我不知道如何解决，希望您百忙之中可以解答我的问题

Can this program predict the situation of intersections?

Hi,
Thank you very much for your excellent work! If I want to apply your work to intersections, what changes do I need to make? Can your achievements be directly applied to the intersections? And I want to predict the traffic situation in the Interaction dataset. What do I need to do? (I'm a beginner, and I don't know the logic at the bottom of the network. Please tell me more, thank you.)
Looking forward to your reply.

Trained models

Hello, thank you very much for the source code for vehicle trajectory prediction. When I unzip the downloaded trained model ,there will always be errors. It has been solved for a long time without success. Can you send me a copy to my email “[email protected]”? thank you.

How to get the ego vehicle's future planning trajectory without ground truth

Nice work. I have a question about the ego vehicle's future planning. In this work, it has been used as input. But for testing and evaluation, how can we get the ego vehicle's future planning trajectory if we do not have the ground truth? It seems that the ego vehicle's future planning trajectory is generated from the ground truth in the code.
Line 325 in data.py:

` def getPlanFuture(self, dsId, planId, refVehId, t):
# Traj of the reference veh
refColIndex = np.where(self.Tracks[dsId - 1][refVehId - 1][0, :] == t)[0][0]
refPos = self.Tracks[dsId - 1][refVehId - 1][1:3, refColIndex].transpose()
# Traj of the planned veh
planColIndex = np.where(self.Tracks[dsId - 1][planId - 1][0, :] == t)[0][0]
stpt = planColIndex
enpt = planColIndex + self.t_f + 1
planGroundTrue = self.Tracks[dsId - 1][planId - 1][1:3, stpt:enpt:self.d_s].transpose()
planFut = planGroundTrue.copy()
# Fitting the downsampled waypoints as the planned trajectory in testing and evaluation.
if self.fit_plan_traj:
wayPoint = np.arange(0, self.t_f + self.d_s, self.d_s)
wayPoint_to_fit = np.arange(0, self.t_f + 1, self.d_s * self.further_ds_plan)
planFut_to_fit = planFut[::self.further_ds_plan, ]
laterParam = fitting_traj_by_qs(wayPoint_to_fit, planFut_to_fit[:, 0])
longiParam = fitting_traj_by_qs(wayPoint_to_fit, planFut_to_fit[:, 1])
planFut[:, 0] = quintic_spline(wayPoint, *laterParam)
planFut[:, 1] = quintic_spline(wayPoint, *longiParam)'

    revPlanFut = np.flip(planFut[1:,] - refPos, axis=0).copy()
    return revPlanFut `

The compressed package of the training model is damaged and cannot be decompressed

In the trained model you provided, the tar file in the downloaded file cannot be decompressed

Sect.4.4中动态图生成代码

您好，请问fig.3和fig.4中的动态图是如何生成的呢？

最近在看你的prime论文可以开源PRIME模型的训练代码么？

数据集下载解压失败

你好，我在你给其他人回复的链接里下载三个数据集但都是解压失败是文件本身上传出错了吗我已经试过好多次了，三个数据集都不行。。。。

about code open-sourcing

Hi,

Thanks for the wonderful work. When is the code coming?

Best,

Usama.

In the file 'preprocess_highD.m', the length of past trajectory is 30 frames and that of future trajectory is 50 frames, which is the same to your implementation in 'preprocess_ngsim.m'. However, as far as I know, the sampling rate of HighD dataset is 25Hz and is different from the sampling rate of NGSIM dataset, which is 10Hz. Could you please share the details about dealing with such problem?
It would be very kind of you if the source code on plotting the figures in your paper will be shared.

haoran-song / pip-planning-informed-prediction Goto Github PK