Perseption module for camera does not work localy. about apollo

[Michael-Fuu]

I want to launch perception module localy and I found this instruction. If you follow it nothing will work. Well, at least it didn't work for me. I use simulation with carla briedge from here.

System information

• OS Platform and Distribution (e.g., Linux Ubuntu 18.04): Linux Ubuntu 22.04
• Apollo installed from (source or binary): source
• Apollo version (3.5, 5.0, 5.5, 6.0): 8.0

Steps to reproduce the issue:

Follow the instruction. You will need to change and add some project files:

1. In file modules/perception/production/conf/perception/camera/fusion_camera_detection_component.pb.txt file change camera_obstacle_perception_conf_file from "camera_detection_pipeline_bev.pb.txt" to "camera_bev_detection_pipeline.pb.txt"
2. From this link download PETR_V1_paddle model and put it on this path modules/perception/production/data/perception/camera/models/yolo_obstacle_detector/
   You now can try to launch your perception.

Supporting materials (screenshots, command lines, code/script snippets):

Full launch log is quite massive, so I'll add only part, that describes the error:

[mainboard_default_67971] I0301 12:17:15.555938 67983 camera_obstacle_detection_component.cc:285] [perception]Enter CameraObstacleDetectionComponent::Proc(), camera_name: front_12mm image ts: 1.70928e+09
[mainboard_default_67971] I0301 12:17:15.556006 67983 camera_obstacle_detection_component.cc:302] [perception]FRAME_STATISTICS:Camera:Start:msg_time[front_12mm-1709284635.515429020]:cur_time[1709284635.556006670]:cur_latency[40.577650070]
[mainboard_default_67971] W0301 12:17:15.556210 67983 buffer.cc:214] [mainboard]BufferCore::canTransform failed: canTransform: target_frame novatel does not exist. canTransform: source_frame front_12mm does not exist.
[mainboard_default_67971] W0301 12:17:15.559298 67983 buffer.cc:214] [mainboard]BufferCore::canTransform failed: canTransform: target_frame novatel does not exist. canTransform: source_frame front_12mm does not exist.
[mainboard_default_67971] W0301 12:17:15.562373 67983 buffer.cc:214] [mainboard]BufferCore::canTransform failed: canTransform: target_frame novatel does not exist. canTransform: source_frame front_12mm does not exist.
[mainboard_default_67971] W0301 12:17:15.565451 67983 buffer.cc:214] [mainboard]BufferCore::canTransform failed: canTransform: target_frame novatel does not exist. canTransform: source_frame front_12mm does not exist.
[mainboard_default_67971] E0301 12:17:15.568578 67983 transform_wrapper.cc:276] [mainboard]Can not find transform. 1709284635.515429020 frame_id: novatel child_frame_id: front_12mm Error info: canTransform: target_frame novatel does not exist. canTransform: source_frame front_12mm does not exist.:timeout
[mainboard_default_67971] E0301 12:17:15.568796 67983 camera_obstacle_detection_component.cc:728] [perception]failed to get camera to world pose, ts: 1.70928e+09 camera_name: front_12mm
[mainboard_default_67971] E0301 12:17:15.568822 67983 camera_obstacle_detection_component.cc:324] [perception]InternalProc failed, error_code: 4001
[mainboard_default_67971] I0301 12:17:15.606477 67986 camera_obstacle_detection_component.cc:285] [perception]Enter CameraObstacleDetectionComponent::Proc(), camera_name: front_12mm image ts: 1.70928e+09
[mainboard_default_67971] I0301 12:17:15.606537 67986 camera_obstacle_detection_component.cc:302] [perception]FRAME_STATISTICS:Camera:Start:msg_time[front_12mm-1709284635.593566179]:cur_time[1709284635.606537580]:cur_latency[12.971401215]
[mainboard_default_67971] W0301 12:17:15.606688 67986 buffer.cc:214] [mainboard]BufferCore::canTransform failed: canTransform: target_frame novatel does not exist. canTransform: source_frame front_12mm does not exist.
[mainboard_default_67971] W0301 12:17:15.609773 67986 buffer.cc:214] [mainboard]BufferCore::canTransform failed: canTransform: target_frame novatel does not exist. canTransform: source_frame front_12mm does not exist.
[mainboard_default_67971] W0301 12:17:15.612855 67986 buffer.cc:214] [mainboard]BufferCore::canTransform failed: canTransform: target_frame novatel does not exist. canTransform: source_frame front_12mm does not exist.
[mainboard_default_67971] W0301 12:17:15.615945 67986 buffer.cc:214] [mainboard]BufferCore::canTransform failed: canTransform: target_frame novatel does not exist. canTransform: source_frame front_12mm does not exist.
[mainboard_default_67971] rm: cannot remove '/apollo/debug_output/*.jpg': No such file or directory
[mainboard_default_67971] I0301 12:17:15.616417 67978 module_controller.cc:65] [mainboard]Start initialize dag: /apollo/modules/perception/production/dag/dag_motion_service.dag
[mainboard_default_67971] I0301 12:17:15.616706 67978 class_loader.cc:37] [mainboard]Begin LoadLibrary: /apollo/bazel-bin/modules/perception/camera/lib/motion_service/libmotion_service.so
[mainboard_default_67971] E0301 12:17:15.619051 67986 transform_wrapper.cc:276] [mainboard]Can not find transform. 1709284635.593566179 frame_id: novatel child_frame_id: front_12mm Error info: canTransform: target_frame novatel does not exist. canTransform: source_frame front_12mm does not exist.:timeout
[mainboard_default_67971] E0301 12:17:15.619098 67986 camera_obstacle_detection_component.cc:728] [perception]failed to get camera to world pose, ts: 1.70928e+09 camera_name: front_12mm
[mainboard_default_67971] E0301 12:17:15.619108 67986 camera_obstacle_detection_component.cc:324] [perception]InternalProc failed, error_code: 4001
[mainboard_default_67971] W0301 12:17:15.637583 67978 class_loader_utility.cc:240] [mainboard]Class factory objs counts is 0, maybe registerclass failed.
[mainboard_default_67971] I0301 12:17:15.640748 67978 motion_service.cc:34] [mainboard]start to init MotionService.
[mainboard_default_67971] mainboard: external/eigen/Eigen/src/Core/DenseStorage.h:128: Eigen::internal::plain_array<T, Size, MatrixOrArrayOptions, 32>::plain_array() [with T = float; int Size = 16; int MatrixOrArrayOptions = 0]: Assertion `(internal::UIntPtr(eigen_unaligned_array_assert_workaround_gcc47(array)) & (31)) == 0 && "this assertion is explained here: " "http://eigen.tuxfamily.org/dox-devel/group__TopicUnalignedArrayAssert.html" " **** READ THIS WEB PAGE !!! ****"' failed.
[cyber_launch_67971] ERROR Process [mainboard_default_67971] has died [pid 67978, exit code -6, cmd mainboard -d /apollo/modules/perception/production/dag/dag_streaming_perception_camera.dag -d /apollo/modules/perception/production/dag/dag_motion_service.dag -p mainboard_default_67971 -s CYBER_DEFAULT].
[cyber_launch_67971] INFO All processes has died.
[cyber_launch_67971] INFO Cyber exit.
[cyber_launch_67971] INFO All processes have been stopped.

I'm 100% sure, that this "front_12mm" camera is presented in the system, because:

1. It is spawn by carla:
   
2. It has an active topic:
   
3. I can connect to it inside dreamview:
   

I don't know if it is connected to this problem in any way, so I'll mention it just in case. After I launch decompression in dreamview I can choose from two /apollo/sensor/camera/front_12mm/image topics despite the fact, that in the system only one remains. Appreciate any help.

I see you choose to use PETR_V1_paddle as the obstacle detection model, which needs six image inputs simultanously.So you might need to change objects.json to spawn 6 rgb cameras.Or you can try to use other obstacle detection model as they most likely only need 6mm rgb camera.