Vision-Language Models (VLMs) in Autonomous Driving (AD) and Intelligent Transportation Systems (ITS) 🚘
This repository collects research papers on Vision-Language Models in Autonomous Driving and Intelligent Transportation Systems. The repo maintained by TUM-AIR will be continuously updated to track the latest work in the community.
If there are any omissions or suggestions, you're warmly welcome to reach out to us ([email protected] or [email protected]).
Keywords: Vision Language Models, Large Language Models, Autonomous Driving, Intelligent Transportation Systems
Please visit Vision Language Models in Autonomous Driving and Intelligent Transportation Systems for more details and comprehensive information. If you find our paper and repo helpful, please consider citing it as follows:
@misc{zhou2023vision,
title={Vision Language Models in Autonomous Driving and Intelligent Transportation Systems},
author={Xingcheng Zhou and Mingyu Liu and Bare Luka Zagar and Ekim Yurtsever and Alois C. Knoll},
year={2023},
eprint={2310.14414},
archivePrefix={arXiv},
primaryClass={cs.CV}
}The applications of Vision-Language Models (VLMs) in the fields of Autonomous Driving (AD) and Intelligent Transportation Systems (ITS) have attracted widespread attention due to their outstanding performance and the ability to leverage Large Language Models (LLMs). By integrating language data, the vehicles, and transportation systems are able to deeply understand real-world environments, improving driving safety and efficiency
| Method | Year | Task | Code Link |
|---|---|---|---|
| Talk to the vehicle: Language conditioned autonomous navigation of self driving car | 2019 | Language-Guided Navigation | |
| Ground then Navigate: Language-guided Navigation in Dynamic Scenes | 2022 | Language-Guided Navigation | |
| ALT-Pilot: Autonomous navigation with Language augmented Topometric maps | 2023 | Vision-Language Localization, Language-Guided Navigation | Page |
| GPT-Driver: Learning to Drive with GPT | 2023 | Motion Planing | Github |
| Can you text what is happening? Integrating pre-trained language encoders into trajectory prediction models for autonomous driving | 2023 | Trajectory Prediction |
| Method | Year | Task | Code Link |
|---|---|---|---|
| Advisable Learning for Self-driving Vehicles by Internalizing Observation-to-Action Rules | 2020 | Open-loop Decision-Making | |
| LanguageMPC: Large Language Models as Decision Makers for Autonomous Driving | 2023 | Open-loop Decision-Making | |
| Receive, Reason, and React: Drive as You Say with Large Language Models in Autonomous Vehicles | 2023 | Open-loop Decision-Making, Motion Planing | |
| Driving with LLMs: Fusing Object-Level Vector Modality for Explainable Autonomous Driving | 2023 | Open-loop Control, Visual Spatial Reasoning | Github |
| DiLu: A Knowledge-Driven Approach to Autonomous Driving with Large Language Models | 2023 | Closed-loop Decision-Making | |
| SurrealDriver: Designing Generative Driver Agent Simulation Framework in Urban Contexts based on Large Language Model | 2023 | Closed-loop Decision-Making | |
| Drive Like a Human: Rethinking Autonomous Driving with Large Language Models | 2023 | Closed-loop Decision-Making | Github |
| Method | Year | Task | Code Link |
|---|---|---|---|
| DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Model | 2023 | Open-loop Control, Visual Question Answering | |
| ADAPT: Action-aware Driving Caption Transformer | 2023 | Open-loop Decision-Making, Visual Spatial Reasoning | Github |
| Method | Year | Task | Code Link |
|---|---|---|---|
| DriveGAN: Towards a Controllable High-Quality Neural Simulation | 2021 | Conditional Video Generation | Page |
| GAIA-1: A Generative World Model for Autonomous Driving | 2023 | Conditional Video Generation | Page |
| DriveDreamer: Towards Real-world-driven World Models for Autonomous Driving | 2023 | Conditional Video Generation | Github |
| DrivingDiffusion: Layout-Guided multi-view driving scene video generation with latent diffusion model | 2023 | Conditional Multi-view Video Generation | Github |
| BEVControl: Accurately Controlling Street-view Elements with Multi-perspective Consistency via BEV Sketch Layout | 2023 | Conditional Image Generation |
| Method | Year | Task | Code Link |
|---|---|---|---|
| A Multi-granularity Retrieval System for Natural Language-based Vehicle Retrieval | 2022 | Language-Guided Vehicle Retrieval | Page |
| Tracked-Vehicle Retrieval by Natural Language Descriptions With Multi-Contextual Adaptive Knowledge | 2023 | Language-Guided Vehicle Retrieval | Page |
| A Unified Multi-modal Structure for Retrieving Tracked Vehicles through Natural Language Descriptions | 2023 | Language-Guided Vehicle Retrieval | Page |
| Traffic-Domain Video Question Answering with Automatic Captioning | 2023 | Image Captioning, Visual Question Answering | |
| Causality-aware Visual Scene Discovery for Cross-Modal Question Reasoning | 2023 | Visual Question Answering | |
| Tem-adapter: Adapting Image-Text Pretraining for Video Question Answer | 2023 | Visual Question Answering | Github |
| Delving into CLIP latent space for Video Anomaly Recognition | 2023 | Video Anomaly Recognition | Github |
| Method | Year | Task | Code Link |
|---|---|---|---|
| LLM Powered Sim-to-real Transfer for Traffic Signal Control | 2023 | Traffic Signal Control |
| Dataset | Year | Task | Data Link |
|---|---|---|---|
| Pedestrian Detection: A Benchmark | 2009 | 2D OD | Link |
| Vision meets robotics: The kitti dataset | 2012 | 2D/3D OD, SS, OT | Link |
| The Cityscapes Dataset for Semantic Urban Scene Understanding | 2016 | 2D/3D OD, SS | Link |
| Citypersons: A diverse dataset for pedestrian detection | 2017 | 2D OD | Link |
| SemanticKITTI: A Dataset for Semantic Scene Understanding of LiDAR Sequences | 2019 | 3D SS | Link |
| Cityflow: A city-scale benchmark for multi-target multi-camera vehicle tracking and re-identification | 2019 | OT, ReID | Link |
| nuscenes: A multimodal dataset for autonomous driving | 2020 | 2D/3D OD, 2D/3D SS, OT, MP | Link |
| BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learning | 2020 | 2D OD, 2D SS, OT | Link |
| Scalability in Perception for Autonomous Driving: Waymo Open Dataset | 2020 | 2D/3D OD, 2D/3D SS, OT | Link |
| Dataset | Year | Task | Data Link |
|---|---|---|---|
| Future Frame Prediction for Anomaly Detection – A New Baseline | 2018 | Anomaly Detection | Link |
| Real-world Anomaly Detection in Surveillance Videos | 2018 | Anomaly Detection | Link |
| Sutd-trafficqa: A question answering benchmark and an efficient network for video reasoning over traffic events | 2021 | Visual Question Answering | Link |
| AerialVLN: Vision-and-Language Navigation for UAVs | 2023 | Vision-Language Navigation | Link |
This repository is released under the Apache 2.0 license.
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent