kin-zhang / mmfn Goto Github PK

View Code? Open in Web Editor NEW

88.0 88.0 8.0 21.71 MB

[IROS'22] MMFN: Multi-Modal Fusion Net for End-to-End Autonomous Driving

License: MIT License

Python 91.37% Ruby 0.07% HTML 1.26% SCSS 0.01% JavaScript 0.07% Shell 0.37% Dockerfile 0.24% CSS 0.13% XSLT 6.49%

mmfn's Introduction

Hi there 👋 , welcome to Kin's GitHub page!

🌱 I'm a Ph.D. student at the DIVISION OF ROBOTICS, PERCEPTION AND LEARNING at KTH Royal Institute of Technology, supervised by Prof. Patric Jensfelt and Prof. Olov Andersson. My research mainly focuses on dynamic awareness in point clouds based on our WASP project. Still exploring...
✨ MPhil Graduated in Aug 2022. I was a MPhil Student at the Robotics Institute RAMLAB of The Hong Kong University of Science and Technology, supervised by Prof. Ming Liu and Prof. Wei Zhang. My MPhil research mainly focused on planning and end-to-end learning for self-driving.
😄 I enjoy learning about different things and techniques, diving into theory, and blogging.
⚡ Life ideal: open-source amazing project, travel around the world

(｡･∀･)ﾉﾞ嗨 👋 , 這裡是張聰明

🌱 現在我是瑞典皇家理工學院的在讀博士，師從Prof. Patric Jensfelt和Prof. Olov Andersson，研究方向：点云中的动态物体感知等工作，還在探索中...
✨ 2022年8月畢業：在劉明教授和張薇教授指導下，於香港科技大學機器人研究所RI RAMLAB 攻讀研究型碩士。碩士期間主要研究無人駕駛中的運動規劃和端到端學習下的自動駕駛系統設計
😄 享受學習不同的知識和技巧，喜歡探究理論部分，同時喜歡寫博客
⚡ 人生理想：想一些有意思idea，然後開源耶✌️，環遊世界

mmfn's People

Contributors

Stargazers

Watchers

Forkers

jonyzhang2023 liuxuanheng jeisonpham avi9700 lhzlhz abol-karimi sailorbrandon

mmfn's Issues

采集数据出现问题

您好，我按照您提供的教程配置了项目运行环境，当我运行phase0_run_eval.py时，出现了以下错误，我尝试为该文件加上可执行权限但并没有解决这个问题。请问该如何解决呢？
Disabling core dumps.
=======> input is a folder, start with for loop
--------------------TEST Agent: auto_pilot.py--------------------
Contiune the route from file: /home/gaoshu/RLmodels/mmfn/data/results/routes_town03_tiny_auto_pilot.json

========= Preparing RouteScenario_31 (repetition 0) =========

Setting up the agent
Data save path /home/gaoshu/RLmodels/mmfn/data/nss_1114/Town03_tiny/Town03_31_11_14_12_45_28
Loading the world
Running the route
Loading success for map, save path is: /home/gaoshu/RLmodels/mmfn/data/nss_1114/Town03_tiny/Town03_31_11_14_12_45_28
start to build rmap. map_number: 1
0%| | 0/1 [00:00<?, ?it/s]/home/gaoshu/RLmodels/mmfn/assets/package/rough_map_node /home/gaoshu/RLmodels/mmfn/data/nss_1114/Town03_tiny/Town03_31_11_14_12_45_28/opendrive
/bin/sh: 1: /home/gaoshu/RLmodels/mmfn/assets/package/rough_map_node: Permission denied

Stopping the route, the agent has crashed:

list index out of range

leaderboard.autoagents.agent_wrapper.AgentError: list index out of range

Stopping the route

========= Results of RouteScenario_0 (repetition 0) ------ FAILURE =========

out of lane

你好，我根据您的代码尝试收集Vector map数据，但是显示out of lane，实际的map.png显示在车道线上，vector得到的值也为0，想问下您有什么好的建议吗

在build_rmap的时候Permission denied

def build_rmap(all_path: list, lib_path):#['/home/BuaaClass02/hrz/mmfn/data/nss_0702/Town02_short/Town02_3_11_23_21_05_32/opendrive'] [tmp_dir] /home/BuaaClass02/hrz/mmfn/assets/package
    print("start to build rmap. map_number:", len(all_path))
    opendrive2vec = os.path.join(lib_path,"rough_map_node")#/home/BuaaClass02/hrz/mmfn/assets/package/rough_map_node
    if not os.path.exists(opendrive2vec):
        print(f"{bc.FAIL}Please install the lib through readme{bc.ENDC}, there will not vector map saved!")
        return True
    is_error = False
    for path in tqdm(all_path):
        cmd2 = opendrive2vec + " " + path#/home/BuaaClass02/hrz/mmfn/assets/package/rough_map_node /home/BuaaClass02/hrz/mmfn/data/nss_0702/Town02_short/Town02_3_11_23_21_05_32/opendrive
        print(cmd2)
        tmp = os.popen(cmd2).readlines()
        # print(tmp[0])
        if (tmp[0][-2:]!="ok"):
            is_error = True
            break

其中在执行tmp = os.popen(cmd2).readlines()的时候报错，bash: /home/BuaaClass02/hrz/mmfn/assets/package/rough_map_node: Permission denied，注意到 #1 这个里面提到是setup_lib的问题，我安装了，但是执行的时候提示Permission denied，应该怎么办呢

安装lib的过程似乎是没有问题的，

(plant_h) BuaaClass02@tsinghua-vm2:~/hrz/mmfn$ ./scripts/setup_lib.sh
Hit:1 http://mirrors.163.com/ubuntu bionic InReleaseHit:2 http://mirrors.163.com/ubuntu bionic-updates InReleaseHit:3 http://mirrors.163.com/ubuntu bionic-backports InRelease
Hit:4 http://mirrors.163.com/ubuntu bionic-security InRelea                                                                            "tsinghua-vm2" 21:10 23-Nov-223 23-Nov-22se
Get:5 https://download.docker.com/linux/ubuntu bionic InRelease [64.4 kB]
Fetched 64.4 kB in 2s (41.2 kB/s)                         
Reading package lists... Done
Building dependency tree
Reading state information... Done
77 packages can be upgraded. Run 'apt list --upgradable' to see them.
Reading package lists... Done
Building dependency tree
Reading state information... Done
protobuf-compiler is already the newest version (3.0.0-9.1ubuntu1).
0 upgraded, 0 newly installed, 0 to remove and 77 not upgraded.
your system is 18.04, cp opendrive to vec lib now...

非常期待您的回答~

About the evaluation

Hello, sorry to bother you, I have some problems in the process of running your source code, I have these errors when I evaluate after training.
--------------------TEST Agent: mmfn_imgnet.py--------------------
It will overwrite the things! ATTENTION /mnt/Data/mmfn/mmfn/data/results/only_one_town_mmfn_imgnet.json

========= Preparing RouteScenario_0 (repetition 0) =========

Setting up the agent
loading model

Could not set up the required agent:

Error(s) in loading state_dict for MMFN:
Missing key(s) in state_dict: "encoder.vectornet_encoder.lane_subgraph.layers.mlp_0.mlp.0.weight", "encoder.vectornet_encoder.lane_subgraph.layers.mlp_0.mlp.0.bias", "encoder.vectornet_encoder.lane_subgraph.layers.mlp_0.mlp.1.weight", "encoder.vectornet_encoder.lane_subgraph.layers.mlp_0.mlp.1.bias", "encoder.vectornet_encoder.lane_subgraph.layers.mlp_1.mlp.0.weight", "encoder.vectornet_encoder.lane_subgraph.layers.mlp_1.mlp.0.bias", "encoder.vectornet_encoder.lane_subgraph.layers.mlp_1.mlp.1.weight", "encoder.vectornet_encoder.lane_subgraph.layers.mlp_1.mlp.1.bias", "encoder.vectornet_encoder.lane_subgraph.layers.mlp_2.mlp.0.weight", "encoder.vectornet_encoder.lane_subgraph.layers.mlp_2.mlp.0.bias", "encoder.vectornet_encoder.lane_subgraph.layers.mlp_2.mlp.1.weight", "encoder.vectornet_encoder.lane_subgraph.layers.mlp_2.mlp.1.bias", "encoder.vectornet_encoder.pos_emb.0.weight", "encoder.vectornet_encoder.pos_emb.0.bias", "encoder.vectornet_encoder.pos_emb.1.weight", "encoder.vectornet_encoder.pos_emb.1.bias", "encoder.vectornet_encoder.pos_emb.3.weight", "encoder.vectornet_encoder.pos_emb.3.bias", "encoder.vectornet_encoder.L2L.to_qkv.weight", "encoder.vectornet_encoder.L2L.to_out.0.weight", "encoder.vectornet_encoder.L2L.to_out.0.bias", "encoder.vectornet_encoder.agent_fusion.0.weight", "encoder.vectornet_encoder.agent_fusion.0.bias", "encoder.vectornet_encoder.agent_fusion.1.weight", "encoder.vectornet_encoder.agent_fusion.1.bias", "encoder.vectornet_encoder.agent_fusion.3.weight", "encoder.vectornet_encoder.agent_fusion.3.bias", "encoder.vectornet_encoder.generator.0.weight", "encoder.vectornet_encoder.generator.0.bias", "encoder.vectornet_encoder.generator.1.weight", "encoder.vectornet_encoder.generator.1.bias", "encoder.vectornet_encoder.generator.3.weight", "encoder.vectornet_encoder.generator.3.bias".

Please install the lib through readme, there will not vector map saved!

张学姐你好，我是一个初入自动驾驶的小萌新研究生。在按照readme进行环境配置后，收集专家数据集时 vector map 无法收集到，尝试了一些修改都没有成功。请学姐多指教！

======== Preparing RouteScenario_1 (repetition 0) =========

Setting up the agent
Data save path /home/drl/TU2023/mmfn/data/nss_0702/Town02_tiny/Town02_1_02_13_17_31_12
Loading the world
Running the route
Loading success for map, save path is: /home/drl/TU2023/mmfn/data/nss_0702/Town02_tiny/Town02_1_02_13_17_31_12
start to build rmap. map_number: 1
Please install the lib through readme, there will not vector map saved!

eval

首先感谢您的贡献，这项工作也非常有趣！然后我想咨询一下您evaluate部分的代码好像和论文所述不太符合，请问有原版的代码吗，十分感谢！

请问如何修改测试时的视角

感谢您的工作，我希望测试时可以显示主驾驶视角或者车前视角，请问我该如何实现。万分感谢。

数据收集时出现错误

感谢大佬的工作，有两个小问题想请教一下大佬：
1、第一个在carla_birdeye_view 里面写到一个路径为/home/kin/birdview_v3_cache/ 这个路径是存在生成鸟瞰图的地址吗，需要怎么修改呢
2、我在收集数据的时候报了以下错误，想请教一下大佬有没有什么建议
Traceback (most recent call last):
File "/home/rao/workspace/ssd/mmfn-main/leaderboard/leaderboard/scenarios/scenario_manager.py", line 152, in _tick_scenario
ego_action = self._agent()
File "/home/rao/workspace/ssd/mmfn-main/leaderboard/leaderboard/autoagents/agent_wrapper.py", line 75, in call
return self._agent()
File "/home/rao/workspace/ssd/mmfn-main/leaderboard/leaderboard/autoagents/autonomous_agent.py", line 115, in call
control = self.run_step(input_data, timestamp)
File "/home/rao/workspace/ssd/mmfn-main/team_code/expert_agent/auto_pilot.py", line 137, in run_step
self.save_map(input_data)
File "/home/rao/workspace/ssd/mmfn-main/team_code/expert_agent/common/base_agent.py", line 171, in save_map
build_rmap([tmp_dir], lib_path)
File "/home/rao/workspace/ssd/mmfn-main/team_code/expert_agent/common/utils.py", line 347, in build_rmap
if (tmp[0][-2:]!="ok"):
IndexError: list index out of range
During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/rao/workspace/ssd/mmfn-main/leaderboard/leaderboard/leaderboard_evaluator.py", line 344, in _load_and_run_scenario
self.manager.run_scenario()
File "/home/rao/workspace/ssd/mmfn-main/leaderboard/leaderboard/scenarios/scenario_manager.py", line 136, in run_scenario
self._tick_scenario(timestamp)
File "/home/rao/workspace/ssd/mmfn-main/leaderboard/leaderboard/scenarios/scenario_manager.py", line 159, in _tick_scenario
raise AgentError(e)
leaderboard.autoagents.agent_wrapper.AgentError: list index out of range

pretrained models

Thanks for the great work, may I know how to download pretrained model and can run eval (at wandb.ai not showing any info)?

Could you provide a trained model img?

Thank you for the great work!
I want to try running mmfn, could you please provide a trained model img for us to download? Also, I am not sure if I can skip the data&train steps to run eval step directly. Could you help me with this?

Python 3.8, CARLA 0.9.14, Leaderboard 2.0

I want to test the trained agent in CARLA 0.9.14. How should I go about it? Do you have any plans to upgrade to Leaderboard 2.0?

typos on readme

git clone https://github.com/Kin-Zhang/mmfn
cd mmfn
conda create -n mmfn python=3.7
-> pip3 install -r requirements.txt
-> conda activate mmfn

: )

Vulkan and OpenGL problem.

Hello Ms.Zhang,

Thanks for your great work, your work is very important to me! I would like to use your work as baseline of my work. But I have a problem when I tried to run your work, it would be a great help if you could help me with this problem!

Here are the details:
I tried to run CarlaUE4.sh on my local machine, it works but the machine gets so slow because the GPU is not good enugh (Nvidia 1030). So I decided to run it on a server with 4 2080Ti cards of my Institute. But the problem is I cannot run CarlaUE4.sh on the serve.
The Error was

and

Does it mean we need to upgrade our graphic card driver or we need to configure something so I can run Carla on server?
Looking forward to see your response!

About collecting data

since official stable leaderboard have memory leak issue. Please check your memory and kill it to resume.
When I run the data-collecting script，I meet the memory leak problem. After the script for collecting data ran for more than three hours, the program terminated and reported an error numpy.core._exceptions._ArrayMemoryError: Unable to allocate 170. MiB for an array with shape (178773021,) and data type uint8. Can I continue running the program to collect data from the checkpoint, or do I have to start the collection from scratch (meaning the previous data is invalid).

About Computing Platform

感谢大佬的工作！

请问您实验使用的GPU型号以及数据采集和训练的耗时，想做一下参考。

Training Data Towns

Hi,

Could you please specify which towns were used to train the model? The train.yaml file has all the towns commented out except for "Town03". Does this mean that only Town03 data was used to generate the results specified in the paper? It would be great if you could specify the final training and validation town used in the paper.

Also the paper mentions 207k frames were used for training. Are these from all 8 CARLA towns? When we ran the data collection script we got lesser data collected (~182k frame).

Your response would be appreciated.

Thanks.