wayveai / driving-with-llms

PyTorch implementation for the paper "Driving with LLMs: Fusing Object-Level Vector Modality for Explainable Autonomous Driving"

This is the PyTorch implementation for inference and training of the LLM-Driver described in:

Driving with LLMs: Fusing Object-Level Vector Modality for Explainable Autonomous Driving

Long Chen, Oleg Sinavski, Jan Hünermann, Alice Karnsund, Andrew James Willmott, Danny Birch, Daniel Maund, Jamie Shotton

ICRA 2024
[preprint] [arxiv]

The LLM-Driver utilises object-level vector input from our driving simulator to predict explanable actions using pretrained Language Models, providing a robust and interpretable solution for autonomous driving.
The LLM-Driver running in open-loop prediction using the vector inputs (top-left BEV view), with the results of action prediction (steering angles and acceleration/brake pedals), action justification (captions on the rendered video), Driving Question Answering (table at the bottom).

News

• [2024/01/29] Thrilled to share that our paper has been accepted by ICRA 2024!
• [2023/12/21] Please checkout our follow-up work LingoQA: [code] [arxiv]
• [2023/10/03] The paper is now avaliable on [arxiv]
• [2023/07/06] The paper and code have been made available under the paper_code branch for anonymous submission.

Getting Started

Prerequisites

• Python 3.x
• pip
• Minimum of 20GB VRAM for running evaluations
• Minimum of 40GB VRAM for training (default setting)

⚙ Setup

1. Set up a virtual environment (tested with Python 3.8-3.11)

   python3 -m venv env
   source env/bin/activate

2. Install required dependencies

   pip install -r requirements.txt.lock

Note: requirements.txt.lock is generated with pip-compile from original requirements.txt for reproducibility.

3. Set up WandB API key

   Set up your WandB API key for training and evaluation logging.

   export WANDB_API_KEY=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

💿 Dataset

• Training/testing data: The datasets have already been checked into the codebase. To unarchive them, use the following commands:

  tar -xzvf data/vqa_train_10k.tar.gz -C data/
  tar -xzvf data/vqa_test_1k.tar.gz -C data/
  

• Re-collect DrivingQA data: While the training and evaluation datasets already include pre-collected DrivingQA data, we also offer a script that illustrates how to collect DrivingQA data using the OpenAI ChatGPT API. If you wish to re-collect the DrivingQA data, simplely run the following command with your OpenAI API key:

  python scripts/collect_vqa.py -i data/vqa_test_1k.pkl -o output_folder/ --openai_api xxxxxxxx

🏄 Evaluation

1. Evaluate for Perception and Action Prediction

   Run the following command:

   python train.py \
       --mode eval \
       --resume_from_checkpoint models/weights/stage2_with_pretrained/ \
       --data_path data/vqa_train_10k.pkl \
       --val_data_path data/vqa_test_1k.pkl \
       --eval_items caption,action \
       --vqa

2. Evaluate for DrivingQA

   Run the following command:

   python train.py \
       --mode eval \
       --resume_from_checkpoint models/weights/stage2_with_pretrained/ \
       --data_path data/vqa_train_10k.pkl \
       --val_data_path data/vqa_test_1k.pkl \
       --eval_items vqa \
       --vqa

3. View Results

   The results can be viewed on the WandB project "llm-driver".

4. Grade DrivingQA Results with GPT API

   To grade the results with GPT API, run the following command:

   python scripts/grade_vqa.py \
       -i data/vqa_test_1k.pkl \
       -o results/10k_ft.pkl \
       -r results/10k_ft.json \
       --openai_api xxxxxxxx

   Replace the results/10k_ft.json with the val_results.table.json downloaded from WandB to grade your results.

🏊 Training

1. Run LLM-Driver Training

   Execute the following command to start training:

   python train.py \
       --mode train \
       --eval_steps 50 \
       --val_set_size 32 \
       --num_epochs 5 \
       --resume_from_checkpoint models/weights/stage1_pretrained_model/ \
       --data_path data/vqa_train_10k.pkl \
       --val_data_path data/vqa_test_1k.pkl \
       --vqa

2. Follow the previous section for evaluating LLM-Driver

3. [optional] Train and evaluate Perceiver-BC

   Execute the following command to start training and evaluation:

   python train_bc.py \
       --num_epochs 25 \
       --data_path data/vqa_train_10k.pkl \
       --val_data_path data/vqa_test_1k.pkl

📝 Citation

If you find our work useful in your research, please consider citing:

@inproceedings{chen2024drivingwithllms,
  title={Driving with LLMs: Fusing Object-Level Vector Modality for Explainable Autonomous Driving},
  author={Long Chen and Oleg Sinavski and Jan Hünermann and Alice Karnsund and Andrew James Willmott and Danny Birch and Daniel Maund and Jamie Shotton},
  booktitle={2024 IEEE International Conference on Robotics and Automation (ICRA)},
  year={2024}
}

@article{marcu2023lingoqa,
  title={LingoQA: Video Question Answering for Autonomous Driving}, 
  author={Ana-Maria Marcu and Long Chen and Jan Hünermann and Alice Karnsund and Benoit Hanotte and Prajwal Chidananda and Saurabh Nair and Vijay Badrinarayanan and Alex Kendall and Jamie Shotton and Oleg Sinavski},
  journal={arXiv preprint arXiv:2312.14115},
  year={2023},
}

🙌 Acknowledgements

This project has drawn inspiration from the Alpaca LoRA repository. We would like to express our appreciation for their contributions to the open-source community.

driving-with-llms's Issues

Dataset and LLM

Hello, your work is very exciting. First of all, I would like to know if your model contains a control signal output. Does the dataset contain a control signal? Second, if I want to replace the LLM with the pre-trained model of the first stage, how can I do it?

The inference generation is very slow

The inference process is currently quite slow. Are there any methods available to accelerate it?
For action task, it costs about 9s for a sample.

RuntimeError: probability tensor contains either `inf`, `nan` or element < 0

Hello, when I evaluate for Perception and Action Prediction, I got this error for decapoda-research/llama-7b-hf.
How can I fix this? Thanks!

Are Driving with LLMs and LingoQA related?

Hello! Amazing work, really.
I was wondering whether this work and the consecutive one (LingoQA) are comparable and if it a real improvement of this work or just another approach.
Since it seems more of a driving scenario commentator and less as a control model that predicts future control signals.
Thank you!

real environment

How to tested in the real environment?

decapoda-research/llama-7b-hf/ Not Found

Thank you for your very nice work!
I was trying to run the program and realized that the pre-trained model is not downloadable from hugging face
decapoda-research/llama-7b-hf/
Can you provide an alternative link for that model?

Paper citations

It is very related to your paper being ICRA 2024 cited, I hope you can have more and better results, but I don't seem to find the citations of this paper on the Internet, all of them are arxiv, do you have a citation in the bib file format of the ICRA2024，Thanks again！

View Results

thanks for your great work!
I want to view the results after evaluation, where to find the WandB project "llm-driver"?

decapoda-research/llama-7b-hf

decapoda-research/llama-7b-hf is no longer accessible on HuggingFace。
I have a bug with “baffo32/decapoda-research-llama-7B-hf”：
ValueError: The device_map provided does not give any device for the following parameters: base_model.model.weighted_mask

RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cuda:1!

File "/home/~~/miniconda3/envs/drive_llm/lib/python3.8/site-packages/transformers/models/llama/modeling_llama.py", line 90, in forward
return self.weight * hidden_states.to(input_dtype)
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cuda:1!

The code and training command of the stage-1: vector representation pre-training stage

Can you provide the code and training command of the first stage: vector representation pre-training stage?

real vehicles

Hi,Thanks for your great work! I would like to inquire, based on your experience and research, do you believe it is feasible to deploy this work onto real vehicles? Particularly, considering our current computational resources are 2 A800 GPUs (80G). In this scenario, how long do you think it might take us to achieve this goal? Are there any key technical challenges or issues that need to be addressed?

Training with real world datasets

Hello, congrats on your work!

I was curious to know if you thought about including the real-world datasets as part of fine-tuning the LLM for decision making?

Unspecified signals

Hi,

I see that the unspecified signals are not used for caption generation. But looks like they were still inputs for the full pipeline.

Would you be able to give the documentation about description of these unspecified signals ?

e.g.https://github.com/wayveai/Driving-with-LLMs/blob/main/utils/vector_utils.py#L111

Have you ever tested in real-world driving environment?

Have you ever tested in real-world driving environment?

How to Use Multi-GPU Training

How to Use Multi-GPU Training

401 Client Error: Unauthorized for url

Hi， the error ‘requests.exceptions.HTTPError: 401 Client Error: Unauthorized for url: https://huggingface.co/decapoda-research/llama-7b-hf/resolve/main/config.json’ occurred wen I run train.py . Whether or not I need add my username and password in the train.py script ?

About the prompt

Driving-with-LLMs/utils/training_utils.py

Line 211 in ae36cc3

{data_point["output"]}"""

Hi, codes here already integrated output label into prompt, so the model already know the real label before prediction? So what about the testing stage when the model don't know the real label?

The environment

I'm grateful for your contributing work. Could you please introduce your setup like CUDA and cudnn? I have tried many times to run the code 'pip install -r requirements.txt.lock' but there are many different problems and errors.

Pre trained model

@olegsinavski @melights @jltorresm @eltociear can u please share thr pretrained model ??

Visualization of the results

@olegsinavski @jltorresm Thanks for sharing the code base can we have the scripts for visualization code also to validate the results

Thanks in advance

cannot download decapoda-research/llama-7b-hf from huggingface

OSError: decapoda-research/llama-7b-hf is not a local folder and is not a valid model identifier listed on 'https://huggingface.co/models'
If this is a private repository, make sure to pass a token having permission to this repo with use_auth_token or log in with huggingface-cli login and pass use_auth_token=True.

Thanks for your work.I have a question. Since the prompt in the model's input already describes the content represented by the vectors, why is it necessary to align the vectors with the LLM during the pre-training process? Are the vectors used to help the model further understand the driving scenario based on the prompt? Are the labels in the pre-training process the prompts generated by lanGen? What is the purpose of the 100k question-answer pairs in the pre-training process?

Could you explain what the model's input and the corresponding labels are during the pre-training stage to help me better understand this process?

does anyone meet 'Parameter' object has no attribute 'CB'?

Hi, I tried to install the requirement file as the author listed but it first raised the error of bitsandbytes not supporting GPU, I solved this by installing bitsandbytes==0.43.3. After that there is an error of 'Parameter' object has no attribute 'CB'. Does anyone have solution?