mczhi / predictive-decision

[ITSC 2023] Predictive Decision-making Framework with Interaction-aware Motion Forecasting Model

Predictive-Decision-making

This repository contains the code for the following paper:

Learning Interaction-aware Motion Prediction Model for Decision-making in Autonomous Driving
Zhiyu Huang, Haochen Liu, Jingda Wu, Wenhui Huang, Chen Lv
AutoMan Research Lab, Nanyang Technological University
[arXiv]

If you are looking for or interested in our winning solutions (Team AID) at NeurIPS 2022 Driving SMARTS Competition, please go to track 1 solution and track 2 solution.

Framework

We propose an interaction-aware predictor to forecast the neighboring agents' future trajectories around the ego vehicle conditioned on the ego vehicle's potential plans. A sampling-based planner will do collision checking and select the optimal trajectory considering the distance to the goal, ride comfort, and safety. The overall framework of our method is given below.

Results

Examples of our framework navigating in various scenarios are shown below.

Left turn

left_turn.mp4

Merge

highway_merge.mp4

Overtake

overtake.mp4

How to use

Create a new Conda environment

conda create -n smarts python=3.8

Install the SMARTS simulator

conda activate smarts

Install the SMARTS simulator.

# Download SMARTS
git clone https://github.com/huawei-noah/SMARTS.git
cd <path/to/SMARTS>
git checkout comp-1

# Install the system requirements.
bash utils/setup/install_deps.sh

# Install smarts with comp-1 branch.
pip install "smarts[camera-obs] @ git+https://github.com/huawei-noah/SMARTS.git@comp-1"

Install Pytorch

conda install pytorch==1.12.0 -c pytorch

Training

Run train.py. Leave other arguments vacant to use the default setting.

python train.py --use_exploration --use_interaction

Testing

Run test.py. You need specify the path to the trained predictor --model_path. You can aslo set --envision_gui to visualize the performance of the framework in envision or set --sumo_gui to visualize in sumo.

python test.py --model_path /training_log/Exp/model.pth

To visualize in Envision (some bugs exist in showing the road map), you need to manually start the envision server and then go to http://localhost:8081/.

scl envision start -p 8081

Citation

If you find this repo to be useful in your research, please consider citing our work

@article{huang2023learning,
  title={Learning Interaction-aware Motion Prediction Model for Decision-making in Autonomous Driving},
  author={Huang, Zhiyu and Liu, Haochen and Wu, Jingda and Huang, Wenhui and Lv, Chen},
  journal={arXiv preprint arXiv:2302.03939},
  year={2023}
}

predictive-decision's Issues

Error comes when testing

Hi Zhiyu, the error comes then testing. Could you pls let me let me know the reason ? Thx a lot.
Running command: python test.py --model_path interaction_aware_predictor.pth

Training is OK.
Running command: python train.py --use_exploration --use_interaction

Questions about the ground truth trajectories of other agents?

Hi MCZhi,

Thank you for sharing such an interesting work. I have some questions about those ground truth trajectories of other agents when training interaction-aware predictor.

1. Do those ground truth trajectories of other agents are generated from log file and so are fixed during training? If so, how does it make sure that other agents will have corresponding reactions on the ego's planning trajectory?

2. What is the difference between the ego-conditioned auto-regressive planning method and traditional imitation learning methods?

Request trained model

Would u pls release the trained model ? Thx a lot.

Questionabout pretrained ckpt

Thank you for your work. I have downloaded the pre-trained model ckpt you provided, but when I tested it, an error occurred. I can ensure that I installed the environment dependencies correctly, so how should I solve this problem?

RuntimeError: Error(s) in loading state_dict for Predictor:
        Unexpected key(s) in state_dict: "decoder.plan_input.weight", "decoder.plan_input.bias", "decoder.state_input.weight", "decoder.state_input.bias".
        size mismatch for decoder.cell.weight_ih: copying a param with shape torch.Size([1152, 128]) from checkpoint, the shape in current model is torch.Size([1152, 3]).

About the result of baseline methods in table II

Hi, thanks for your work!

I have some questions about the success rate about the baseline method in table II.
I have used the pretrained model provided in the repo for the test.py, but the success rate of "Ours" methods is only 80% for the first scene, which is lower than the 98% in the paper.

My command line is:
nohup python test.py --name baseline --use_interaction --model_path PATH/interaction_aware_predictor.pth

Did I make any mistakes in my reproduction steps?

question about scenario

Hello, Zhiyu,
Thank you for your work,
I would like to ask if during the training session, the code will report an error 'Environment multi scenario does not exist.'
May I ask how to solve it.
Thank you!

Some questions about the trajectory generation code

I read the source code and found that the input obs.waypoint_paths of the planner already contains 51 waypoints. The trajectory sampling part will first use the interpolation method to form the route formed by these waypoints into a CubicSpline2D curve x=f(s),y=f( s), and then generate_trajectory_on_frenet will generate various trajectories giving different accelerations (due to the difference in acceleration, some new trajectories will indeed be generated). But I found the calculate_global_trajectory function still uses the CubicSpline2D curve formed by route to calculate fx and fy, which means that even if many trajectories are generated , but the waypoints from the route still need to be passed through in theory. So if I understand correctly, the waypoint provided by obs.waypoint_paths is actually similar to the target point of the carla simulator. I'm wondering if there's something wrong with my understanding. And if this is the case, can it be possible to use only two waypoints as the starting point and end point, and then generate more diverse trajectories?
In addition, I don't quite understand why calculate_global_trajectory uses CubicSpline2D's i_yaw based on the original route to calculate fx and fy.
I'm very sorry if there is any misunderstanding, I am not very familiar with the smarts simulator.