Comments (7)
the occupancy task seems to complement the 3D detection and related prediction tasks, such as abnormal object detection and corner case detection.
@haibao-yu Yes, this is also one of our motivations for introducing occupancy in UniAD. Additionally, the occupancy is per-grid supervised (supervise on the whole BEV plane), much denser than the training of detection (supervise on some sparse vectors in low dimension). Therefore, we think the resulting per-grid probability of occupancy is more reliable than that of detection results. For example, in detection results, a small digital error in the position/heading angle of an output vector would lead to significant deviations from the real-world state.
However, it appears that in UniAD, BEV occupancy can be directly obtained through detection and motion forecasting.
Yes, you can actually extract the future occupancy via some post-progresses on the detection and motion results. However, both detection and motion are trained with sparse supervision on low-dimension vectors, where a small digital error would lead to significant deviations in real world as I mentioned before. By combining their predictions together to obtain occupancy, their errors will be further accumulated
That's why we directly learn occupancy with per-grid dense supervision.
from uniad.
Thank you for your response. Based on the information provided, it seems that there is no direct connection between the usage of the Occupancy labels in UniAD and the Occupancy challenge described at https://github.com/CVPR2023-3D-Occupancy-Prediction/CVPR2023-3D-Occupancy-Prediction. Additionally, I am interested to know if BEV Occupancy is considered a sub-task of BEV detection in UniAD, considering that the occupancy outputs focuses solely on vehicle boxes.
from uniad.
Hi, @haibao-yu. Please refer to this file for occupancy ground-truth generation: https://github.com/OpenDriveLab/UniAD/blob/main/projects/mmdet3d_plugin/datasets/pipelines/occflow_label.py.
Briefly speaking, we draw the annotated 3d boxes of vehicles on a bev plane while ignoring the height of boxes. Pixels within boxes are filled with a "1" to indicate occupied regions, otherwise filled with "0" to indicate empty space.
As each ground-truth is a sequence of multi-frame future occupancy, the coordinates of all frames in the sequence should be transformed to align with the first frame.
from uniad.
@haibao-yu Yes, the 3d occupancy described in https://github.com/CVPR2023-3D-Occupancy-Prediction/CVPR2023-3D-Occupancy-Prediction has major differences with BEV occupancy in UniAD.
- The 3d occupancy challenge considers both dynamic agents and static elements in the scene, as long as there're lidar points on these objects. However, in BEV occupancy, only vehicles are emphasized without the awareness of static objects (trees, buildings).
- UniAD predicts multi-step future BEV occupancies while 3d occupancy challenge only requires predicting the result of the curent timestep.
BEV occupancy also varies from detection:
- Output format: BEV occupancy is similar to 2d segmentation (of a BEV plane), while detections are collections of sparse attributes, indicating both spatial and dynamic information.
- Timestep: BEV occupancy -- Multi-step in the future, Detection -- Single-step of current state
Due to these differences, we do not consider BEV occupancy prediction as a sub-task of detection, and the model designs of these two tasks are different.
from uniad.
Thank you for your quick response.
Regarding the second concern about the relationship between BEV occupancy and BEV detection, I would like to explore the inclusion of motion forecasting information. Specifically, when we combine the detection frame, how does the predicted future trajectory of the dynamic target differ from the concept of occupancy in this context?
The reason I am interested in this question is that, in my perspective, the occupancy task seems to complement the 3D detection and related prediction tasks, such as abnormal object detection and corner case detection. However, it appears that in UniAD, BEV occupancy can be directly obtained through detection and motion forecasting.
from uniad.
the occupancy task seems to complement the 3D detection and related prediction tasks, such as abnormal object detection and corner case detection.
@haibao-yu Yes, this is also one of our motivations for introducing occupancy in UniAD. Additionally, the occupancy is per-grid supervised (supervise on the whole BEV plane), much denser than the training of detection (supervise on some sparse vectors in low dimension). Therefore, we think the resulting per-grid probability of occupancy is more reliable than that of detection results. For example, in detection results, a small digital error in the position/heading angle of an output vector would lead to significant deviations from the real-world state.
However, it appears that in UniAD, BEV occupancy can be directly obtained through detection and motion forecasting.
Yes, you can actually extract the future occupancy via some post-progresses on the detection and motion results. However, both detection and motion are trained with sparse supervision on low-dimension vectors, where a small digital error would lead to significant deviations in real world as I mentioned before. By combining their predictions together to obtain occupancy, their errors will be further accumulated E=Etrack⋅Emotion, making the results more unreliable.
That's why we directly learn occupancy with per-grid dense supervision.
Very thanks for your response.
from uniad.
Thanks for your great job!
What are the connections and differences between occupancy prediction and occupancy flow prediction? Which one is employed in your paper?
Additionally, could you recommend some papers on BEV occupancy prediction?
Thanks again! 谢谢~
from uniad.
Related Issues (20)
- Why is the queue_length set to 5 and 3 HOT 3
- 大佬们 有在实车上跑过的吗 HOT 1
- MMCV Compatibility Issues When Setting Up The Environment on AGX Orin
- Question: how to run the model on a demo video
- AssertionError: MMCV==1.4.0 is used but incompatible. Please install mmcv>=2.0.0rc4, <2.2.0. HOT 1
- Question: How to extend the forecast time
- deploying
- Error during train stage2_e2e HOT 2
- Evaluation about uniad_base_e2e.pth HOT 1
- GPU memory is not released after training process is stopped HOT 2
- WARNING!!!!, Only can be used for obtain inference speed!!!! load checkpoint from local path: ./ckpts/uniad_base_track_map.pth The model and loaded state dict do not match exactly HOT 1
- 执行测试的时候出现模型权重的节点无法完全匹配 HOT 2
- Function _load_future_anns_plan is not defined HOT 2
- batch size set
- messi bailando HOT 1
- GPU and Memory HOT 2
- Occupany Prediction
- project itself and mmdet3d numpy incompatible ; predict demo vedio completely different from the official one
- RuntimeError: replicas[0][0] in this process with sizes [200, 128] appears not to match sizes of the same param in process 0.
- TypeError: expected str, bytes or os.PathLike object, not _io.BufferedReader HOT 4
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from uniad.