cxxpython9 / openlane-v2

OpenLane-V2: The World's First Perception and Reasoning Benchmark for Scene Structure in Autonomous Driving

OpenLane-V2

The World's First Perception and Reasoning Benchmark for Scene Structure in Autonomous Driving.

English | 中文

In terms of ambiguity, the English version shall prevail.

Table of Contents

• News
• Benchmark and Leaderboard
• Highlight
• Task
  • 3D Lane Detection 🛣️
  • Traffic Element Recognition 🚥
  • Topology Recognition 🕸️
• Data
• Devkit
• Get Started
• Train a Model
• Citation
• License

News

• [2023/04]
  • A strong baseline based on InternImage released. Check out here. 🚀
  • OpenLane-V2 paper is available on arXiv.
  • We reiterated the rules of the Challenge:
    • During inference, the input modality of the model should be camera only;
    • No future frame is allowed during inference.
  • A stronger baseline released. Check out here.
• [2023/03]
  • We are hosting a Challenge at the CVPR 2023 Workshop. ⭐
• [2023/02]
  • Dataset v1.0: Data of subset_A released.
  • Baseline model released.
• [2023/01]
  • Dataset v0.1: Initial OpenLane-V2 dataset sample released.
  • Devkit v0.1.0: Initial OpenLane-V2 devkit released.

(back to top)

Benchmark and Leaderboard

We will provide an initial benchmark on the OpenLane-V2 dataset, please stay tuned for the release.

Currently, we are maintaining leaderboards on the val and test split of subset_A.

(back to top)

Highlight - why we are exclusive?

The world is three-dimensional - Introducing 3D lane

Previous datasets annotate lanes on images in the perspective view. Such a type of 2D annotation is insufficient to fulfill real-world requirements. Following the OpenLane dataset, we annotate lanes in 3D space to reflect their properties in the real world.

Be aware of traffic signals - Recognizing Extremely Small road elements

Not only preventing collision but also facilitating efficiency is essential. Vehicles follow predefined traffic rules for self-disciplining and cooperating with others to ensure a safe and efficient traffic system. Traffic elements on the roads, such as traffic lights and road signs, provide practical and real-time information.

Beyond perception - Topology Reasoning between lane and road elements

A traffic element is only valid for its corresponding lanes. Following the wrong signals would be catastrophic. Also, lanes have their predecessors and successors to build the map. Autonomous vehicles are required to reason about the topology relationships to drive in the right way. In this dataset, we hope to shed light on the task of scene structure perception and reasoning.

Data scale and diversity matters - building on Top of Awesome Benchmarks

Experience from the sunny day does not apply to the dancing snowflakes. For machine learning, data is the must-have food. We provide annotations on data collected in various cities, from Austin to Singapore and from Boston to Miami. The diversity of data enables models to generalize in different atmospheres and landscapes.

(back to top)

Task

The primary task of the dataset is scene structure perception and reasoning, which requires the model to recognize the dynamic drivable states of lanes in the surrounding environment. The challenge of this dataset includes not only detecting lane centerlines and traffic elements but also recognizing the attribute of traffic elements and topology relationships on detected objects. We define the OpenLane-V2 Score (OLS), which is the average of various metrics covering different aspects of the primary task:

$$ \text{OLS} = \frac{1}{4} \bigg[ \text{DET}{l} + \text{DET}{t} + f(\text{TOP}{ll}) + f(\text{TOP}{lt}) \bigg]. $$

The metrics of different subtasks are described below.

3D Lane Detection 🛣️

The OpenLane dataset, which is the first real-world and the largest scaled 3D lane dataset to date, provides lane line annotations in 3D space. Similarly, we annotate 3D lane centerlines and include the F-Score for evaluating predicted results of undirected lane centerlines. Furthermore, we define the subtask of 3D lane detection as detecting directed 3D lane centerlines from the given multi-view images covering the whole horizontal FOV. The instance-level evaluation metric of average precision $\text{DET}_{l}$ is utilized to measure the detection performance on lane centerlines (l).

Traffic Element Recognition 🚥

Traffic elements and their attribute provide crucial information for autonomous vehicles. The attribute represents the semantic meaning of a traffic element, such as the red color of a traffic light. In this subtask, on the given image in the front view, the location of traffic elements (traffic lights and road signs) and their attributes are demanded to be perceived simultaneously. Compared to typical 2D detection datasets, the challenge is that the size of traffic elements is tiny due to the large scale of outdoor environments. Similar to the typical 2D detection task, the metric of $\text{DET}_{t}$ is utilized to measure the performance of traffic elements (t) detection averaged over different attributes.

Topology Recognition 🕸️

We first define the task of recognizing topology relationships in the field of autonomous driving. Given multi-view images, the model learns to recognize the topology relationships among lane centerlines and between lane centerlines and traffic elements. The most similar task is link prediction in the field of graph, in which the vertices are given and only edges are predicted by models. In our case, both vertices and edges are unknown for the model. Thus, lane centerlines and traffic elements are needed to be detected first, and then the topology relationships are built. Adapted from the task of link prediction, $\text{TOP}$ is used for topology among lane centerlines (ll) and between lane centerlines and traffic elements (lt).

(back to top)

Data

The OpenLane-V2 dataset is a large-scale dataset for scene structure perception and reasoning in the field of autonomous driving. Following OpenLane, the first 3D lane dataset, we provide lane annotations in 3D space. The difference is that instead of lane lines, we annotate lane centerlines, which can be served as the trajectory for autonomous vehicles. Besides, we provide annotations on traffic elements (traffic lights and road signs) and their attribute, and the topology relationships among lane centerlines and between lane centerlines and traffic elements.

The dataset is divided into two subsets. The subset_A serves as the primary subset and is utilized for the coming challenges and leaderboard, in which no external data, including the other subset, is allowed. The subset_B can be used to test the generalization ability of the model. For more details, please refer to the corresponding pages: use of data, notes of annotation, and dataset statistics.

Download now to discover our dataset!

(back to top)

Devkit

We provide a devkit for easy access to the OpenLane-V2 dataset. After installing the package, the use of the dataset, such as loading images, loading meta data, and evaluating results, can be accessed through the API of openlanv2. For more details on the API, please refer to devkit.

(back to top)

Get Started

Please follow the steps below to get familiar with the OpenLane-V2 dataset.

1. Run the following commands to install the environment for setting up the dataset:

   git clone https://github.com/OpenDriveLab/OpenLane-V2.git
   cd OpenLane-V2
   conda create -n openlanev2 python=3.8 -y
   conda activate openlanev2
   pip install -r requirements.txt
   python setup.py develop

2. Use links to download data manually from

   • OpenDataLab,
   • Google Drive,
   • Baidu Yun.

   Then put them into the data/OpenLane-V2/ folder and unzip them. The resulting folder hierarchy is described here. Or use the following commands to download example data for a quick glance at the dataset:

   cd data/OpenLane-V2
   wget --load-cookies /tmp/cookies.txt "https://docs.google.com/uc?export=download&confirm=$(wget --quiet --save-cookies /tmp/cookies.txt --keep-session-cookies --no-check-certificate 'https://docs.google.com/uc?export=download&id=1Ni-L6u1MGKJRAfUXm39PdBIxdk_ntdc6' -O- | sed -rn 's/.*confirm=([0-9A-Za-z_]+).*/\1\n/p')&id=1Ni-L6u1MGKJRAfUXm39PdBIxdk_ntdc6" -O OpenLane-V2_sample.tar
   md5sum -c openlanev2.md5
   tar -xvf *.tar
   cd ../..

3. Run the tutorial on jupyter notebook to get familiar with the dataset and devkit.

(back to top)

Train a Model

Plug-ins to prevail deep learning frameworks for training models are provided to start training models on the OpenLane-V2 dataset. We appreciate your valuable feedback and contributions to plug-ins on different frameworks.

mmdet3d

The plug-in to MMDetection3d is built on top of mmdet3d v1.0.0rc6 and tested under:

• Python 3.8.15
• PyTorch 1.9.1
• CUDA 11.1
• GCC 5.4.0
• mmcv-full==1.5.2
• mmdet==2.26.0
• mmsegmentation==0.29.1

Please follow the instruction to install mmdet3d. Assuming OpenLane-V2 is installed under OpenLane-V2/ and mmdet3d is built under mmdetection3d/, create a soft link to the plug-in file:

└── mmdetection3d
    └── projects
        ├── example_project
        └── openlanev2 -> OpenLane-V2/plugin/mmdet3d

Then you can train or evaluate a model using the config mmdetection3d/projects/openlanev2/configs/baseline.py, whose path is replaced accordingly. Options can be passed to enable supported functions during evaluation (--eval *), such as --eval-options dump=True dump_dir=/PATH/TO/DUMP to save pickle file for submission and --eval-options visualization=True visualization_dir=/PATH/TO/VIS for visualization.

(back to top)

Citation

Please use the following citation when referencing OpenLane-V2:

@article{wang2023openlanev2,
  title={Road Genome: A Topology Reasoning Benchmark for Scene Understanding in Autonomous Driving}, 
  author={Wang, Huijie and Liu, Zhenbo and Li, Yang and Li, Tianyu and Chen, Li and Sima, Chonghao and Wang, Yuting and Jiang, Shengyin and Wen, Feng and Xu, Hang and Luo, Ping and Yan, Junchi and Zhang, Wei and Yao, Jun and Qiao, Yu and Li, Hongyang},
  journal={arXiv preprint arXiv:2304.10440},
  year={2023}
}

(back to top)

License

Our dataset is built on top of the nuScenes and Argoverse 2 datasets. Before using the OpenLane-V2 dataset, you should agree to the terms of use of the nuScenes and Argoverse 2 datasets respectively. All code within this repository is under Apache License 2.0.

(back to top)