intel / ros2_openvino_toolkit

ros2_openvino_toolkit

Table of Contents

• ➤ Overview
  • ROS2 Version Supported
  • Inference Features Supported
• ➤ Prerequisite
• ➤ Introduction
  • Design Architecture
  • Logic Flow
• ➤ Supported Features
  • Multiple Input Components
  • Inference Implementations
  • ROS Interfaces and Outputs
  • Demo Result Snapshots
• ➤ Installation & Launching
  • Deploy in Local Environment
  • Deploy in Docker
• ➤ Reference
• ➤ FAQ
• ➤ Feedback
• ➤ More Information

Overview

ROS2 Version Supported

Branch Name	ROS2 Version Supported	Openvino Version	OS Version
ros2	Galactic, Foxy, Humble	V2022.1, V2022.2, V2022.3	Ubuntu 20.04, Ubuntu 22.04
dashing	Dashing	V2022.1, V2022.2, V2022.3	Ubuntu 18.04
foxy-ov2021.4	Foxy	V2021.4	Ubuntu 20.04
galactic-ov2021.4	Galactic	V2021.4	Ubuntu 20.04

Inference Features Supported

• Object Detection
• Face Detection
• Age Gender Recognition
• Emotion Recognition
• Head Pose Estimation
• Object Segmentation
• Person Re-Identification
• Vehicle Attribute Detection
• Vehicle License Plate Detection

Prerequisite

Prerequisite	Mandatory?	Description
Processor	Mandatory	A platform with Intel processors assembled. (Refer to here for the full list of Intel processors supported.)
OS	Mandatory	We only tested this project under Ubuntu distros. It is recommended to install the corresponding Ubuntu Distro according to the ROS distro that you select to use. For example: Ubuntu 18.04 for dashing, Ubuntu 20.04 for Foxy and Galactic, Ubuntu 22.04 for Humble.
ROS2	Mandatory	We have already supported active ROS distros (Humble, Galactic, Foxy and Dashing (deprecated)). Choose the one matching your needs. You may find the corresponding branch from the table above in section ROS2 Version Supported.
OpenVINO	Mandatory	The version of OpenVINO toolkit is decided by the OS and ROS2 distros you use. See the table above in Section ROS2 Version Supported.
Realsense Camera	Optional	Realsense Camera is optional, you may choose these alternatives as the input: Standard Camera, ROS Image Topic, Video/Image File or RTSP camera.

Introduction

Design Architecture

From the view of hirarchical architecture design, the package is divided into different functional components, as shown in below picture.

Intel® OpenVINO™ toolkit

• Intel® OpenVINO™ toolkit provides a ROS-adapted runtime framework of neural network which quickly deploys applications and solutions for vision inference. By leveraging Intel® OpenVINO™ toolkit and corresponding libraries, this ROS2 runtime framework extends workloads across Intel® hardware (including accelerators) and maximizes performance.
  • Increase deep learning workload performance up to 19x1 with computer vision accelerators from Intel.
  • Unleash convolutional neural network (CNN)-based deep learning inference using a common API.
  • Speed development using optimized OpenCV* and OpenVX* functions. See more from here for Intel OpenVINO™ introduction.

ROS OpenVINO Runtime Framework

• ROS OpenVINO Runtime Framework is the main body of this repo. It provides key logic implementation for pipeline lifecycle management, resource management and ROS system adapter, which extends Intel OpenVINO toolkit and libraries. Furthermore, this runtime framework provides ways to simplify launching, configuration, data analysis and re-use.

ROS Input & Output

• Diversal Input resources are data resources to be infered and analyzed with the OpenVINO framework.
• ROS interfaces and outputs currently include Topic and service. Natively, RViz output and CV image window output are also supported by refactoring topic message and inferrence results.

Optimized Models

• Optimized Models provided by Model Optimizer component of Intel® OpenVINO™ toolkit. Imports trained models from various frameworks (Caffe*, Tensorflow*, MxNet*, ONNX*, Kaldi*) and converts them to a unified intermediate representation file. It also optimizes topologies through node merging, horizontal fusion, eliminating batch normalization, and quantization. It also supports graph freeze and graph summarize along with dynamic input freezing.

Logic Flow

From the view of logic implementation, the package introduces the definitions of parameter manager, pipeline and pipeline manager. The following picture depicts how these entities co-work together when the corresponding program is launched.

Once a corresponding program is launched with a specified .yaml config file passed in the .launch file or via commandline, parameter manager analyzes the configurations about pipeline and the whole framework, then shares the parsed configuration information with pipeline procedure. A pipeline instance is created by following the configuration info and is added into pipeline manager for lifecycle control and inference action triggering.

The contents in .yaml config file should be well structured and follow the supported rules and entity names. Please see yaml configuration guidance for how to create or edit the config files.

Pipeline

Pipeline fulfills the whole data handling process: initiliazing Input Component for image data gathering and formating; building up the structured inference network and passing the formatted data through the inference network; transfering the inference results and handling output, etc.

Pipeline manager

Pipeline manager manages all the created pipelines according to the inference requests or external demands (say, system exception, resource limitation, or end user's operation). Because of co-working with resource management and being aware of the whole framework, it covers the ability of performance optimization by sharing system resource between pipelines and reducing the burden of data copy.

Supported Features

Multiple Input Components

Currently, the package supports several input resources for acquiring image data. The following tables are listed:

Input Resource Table

Input Resource	Description
StandardCamera	Any RGB camera with USB port supporting. Currently only the first USB camera if many are connected.
RealSenseCamera	Intel RealSense RGB-D Camera, directly calling RealSense Camera via librealsense plugin of openCV.
ImageTopic	Any ROS topic which is structured in image message.
Image	Any image file which can be parsed by openCV, such as .png, .jpeg.
Video	Any video file which can be parsed by openCV.
IpCamera	Any RTSP server which can push video stream.

Inference Implementations

Currently, the corresponding relation of supported inference features, models used and yaml configurations are listed as follows:

Inference Feature Correspondence Table

Inference	Description	YAML Configuration	Model Used
Face Detection	Object Detection task applied to face recognition using a sequence of neural networks.	pipeline_image.yaml pipeline_image_video.yaml pipeline_people.yaml pipeline_people_ip.yaml	face-detection-adas-0001 age-gender-recognition-retail-0013 emotions-recognition-retail-0003 head-pose-estimation-adas-0001
Emotion Recognition	Emotion recognition based on detected face image.	pipeline_image.yaml pipeline_image_video.yaml pipeline_people.yaml pipeline_people_ip.yaml	emotions-recognition-retail-0003
Age & Gender Recognition	Age and gender recognition based on detected face image.	pipeline_image.yaml pipeline_image_video.yaml pipeline_people.yaml pipeline_people_ip.yaml	age-gender-recognition-retail-0013
Head Pose Estimation	Head pose estimation based on detected face image.	pipeline_image.yaml pipeline_image_video.yaml pipeline_people.yaml pipeline_people_ip.yaml	head-pose-estimation-adas-0001
Object Detection	Object detection based on SSD-based trained models.	pipeline_object.yaml pipeline_object_topic.yaml	mobilenet-ssd
Vehicle and License Detection	Vehicle and license detection based on Intel models.	pipeline_vehicle_detection.yaml	vehicle-license-plate-detection-barrier-0106 vehicle-attributes-recognition-barrier-0039 license-plate-recognition-barrier-0001
Object Segmentation	Object segmentation.	pipeline_segmentation.yaml pipeline_segmentation_image.yaml pipeline_video.yaml	semantic-segmentation-adas-0001 deeplabv3
Person Attributes	Person attributes based on object detection.	pipeline_person_attributes.yaml	person-attributes-recognition-crossroad-0230 person-detection-retail-0013
Person Reidentification	Person reidentification based on object detection.	pipeline_person_reidentification.yaml	person-detection-retail-0013 person-reidentification-retail-0277
Object Segmentation Maskrcnn	Object segmentation and detection based on maskrcnn model.	pipeline_segmentation_maskrcnn.yaml	mask_rcnn_inception_v2_coco_2018_01_28

ROS interfaces and outputs

The inference results can be output in several types. One or more types can be enabled for any inference pipeline.

Topic

Specific topic(s) can be generated and published according to the given inference functionalities.

Published Topic Correspondence Table

Inference	Published Topic
People Detection	/ros2_openvino_toolkit/face_detection(object_msgs:msg:ObjectsInBoxes)
Emotion Recognition	/ros2_openvino_toolkit/emotions_recognition(object_msgs:msg:EmotionsStamped)
Age and Gender Recognition	/ros2_openvino_toolkit/age_genders_Recognition(object_msgs:msg:AgeGenderStamped)
Head Pose Estimation	/ros2_openvino_toolkit/headposes_estimation(object_msgs:msg:HeadPoseStamped)
Object Detection	/ros2_openvino_toolkit/detected_objects(object_msgs::msg::ObjectsInBoxes)
Object Segmentation	/ros2_openvino_toolkit/segmented_obejcts(object_msgs::msg::ObjectsInMasks)
Object Segmentation Maskrcnn	/ros2_openvino_toolkit/segmented_obejcts(object_msgs::msg::ObjectsInMasks)
Person Reidentification	/ros2_openvino_toolkit/reidentified_persons(object_msgs::msg::ReidentificationStamped)
Vehicle Detection	/ros2_openvino_toolkit/detected_vehicles_attribs(object_msgs::msg::VehicleAttribsStamped)
Vehicle License Detection	/ros2_openvino_toolkit/detected_license_plates(object_msgs::msg::LicensePlateStamped)

Service

Several ROS2 Services are created, expecting to be used in client/server mode, especially when synchronously getting inference results for a given image frame or when managing inference pipeline's lifecycle.

Service Correspondence Table

Inference	Service
Object Detection Service	/detect_object(object_msgs::srv::DetectObject)
Face Detection Service	/detect_face(object_msgs::srv::DetectObject)
Age Gender Detection Service	/detect_age_gender(object_msgs::srv::AgeGender)
Headpose Detection Service	/detect_head_pose(object_msgs::srv::HeadPose)
Emotion Detection Service	/detect_emotion(object_msgs::srv::Emotion)

RViz

RViz display is also supported by the composited topic of original image frame with inference result. To show in RViz tool, add an image marker with the composited topic: /ros2_openvino_toolkit/image_rviz(sensor_msgs::Image)

Image Window

OpenCV based image window is natively supported by the package. To enable window, Image Window output should be added into the output choices in .yaml config file. Refer to the config file guidance for more information about checking/adding this feature in your launching.

Demo Result Snapshots

For the snapshot of demo results, refer to the following picture.

• Face detection input from standard camera

• Object detection input from realsense camera

• Object segmentation input from video

• Person reidentification input from standard camera

Installation and Launching

Deploy in Local Environment

• Refer to the quick start document for getting_started_with_ros2 for detailed installation & lauching instructions.
• Refer to the quick start document for yaml configuration guidance for detailed configuration guidance.

Deploy in Docker

• Refer to the docker instruction for docker_instructions for detailed information about building docker image and launching.
• Refer to the quick start document for yaml configuration guidance for detailed configuration guidance.

Reference

• Open_model_zoo: Refer to the OpenVINO document for open_model_zoo for detailed model structure and demo samples.
• OpenVINO api 2.0: Refer to the OpenVINO document for OpenVINO_api_2.0 for latest api 2.0 transition guide.

FAQ

• How to get the IR file for yolov5?
• How to build OpenVINO by source?
• How to build RealSense by source?
• What is the basic command of Docker CLI?
• What is the canonical C++ API for interacting with ROS?

Feedback

• Report questions, issues and suggestions, using: issue.

More Information

• ROS2 OpenVINO discription written in Chinese: https://mp.weixin.qq.com/s/BgG3RGauv5pmHzV_hkVAdw

Any security issue should be reported using process at https://01.org/security

ros2_openvino_toolkit's Issues

run multiple pipeline failed

When I tried to launch multiple pipeline with StandardCamera and RealSenseCamera, the RealSenseCamera works fine, but the StandardCamera doesn't work.

Mixed ros crystal and dashing version in environment_setup.sh on dashing branch

for dashing branch, 109 line in environment_setup.sh is as bellows:

 rosdep install --from-paths src --ignore-src --rosdistro crystal -y --skip-keys "console_bridge fastcdr fastrtps libopensplice67 libopensplice69 rti-connext-dds-5.3.1 urdfdom_headers"

Should we install dashing version here?

Running environment_setup.sh

I get the following error with devel version:
OK ./environment_setup.sh: line 80: http_proxy: unbound variable

timestamp in output topic is inconsistent with the corresponding input topic

I print some timestamps in pipeline.cpp and find that the timestamp is inconsistent with the one for the frame being processed.

 auto test_t0 = rclcpp::Clock(RCL_ROS_TIME).now();
  std::unique_lock<std::mutex> lock(counter_mutex_);
  cv_.wait(lock, [self = this]() {return self->counter_ == 0;});
  
  auto headerr = getInputDevice()->getHeader();
  auto msg_time = rclcpp::Time(headerr.stamp);
  auto test_t1 = rclcpp::Clock(RCL_ROS_TIME).now();
  std::cout << "**************************" << std::endl;
  std::cout << "msg_time: " << msg_time.nanoseconds() << std::endl;
  std::cout << "test_t0: " << test_t0.nanoseconds() << std::endl;
  std::cout << "test_t1: " << test_t1.nanoseconds() << std::endl;
  std::cout << "t1-t0: " << (test_t1-test_t0).nanoseconds()*1e-6 << std::endl;
  std::cout << "msg duration: " << (test_t1 - msg_time).nanoseconds()*1e-6 << std::endl;
  std::cout << "**************************" << std::endl;

Output:

[pipeline_with_params-1] **************************
[pipeline_with_params-1] msg_time: 1572420170529247967
[pipeline_with_params-1] test_t0: 1572420170502623887
[pipeline_with_params-1] test_t1: 1572420170547616668
[pipeline_with_params-1] t1-t0: 44.9928
[pipeline_with_params-1] msg duration: 18.3687
[pipeline_with_params-1] **************************

test_t0 is even smaller than msg_time, which is abnormal. This issue may impact performance test.

The problem in dynamic_vino_sample/CMakeList.txt

hi,

when I try:
colcon build --symlink-install
the file cannot find error would be reported :

(My environment is :
Ubuntu 18.04
openvino 2019_R3.1)

It seems like that the include path has problems, so when checking dynamic_vino_sample/CMakeList.txt ,I found this part:

include_directories( # ${CMAKE_CURRENT_SOURCE_DIR}/common/format_reader ${CMAKE_CURRENT_SOURCE_DIR}/include ${InferenceEngine_INCLUDE_DIRS} ${InferenceEngine_INCLUDE_DIRS}/../samples ${InferenceEngine_INCLUDE_DIRS}/../samples/extention ("extension"?) ${InferenceEngine_INCLUDE_DIRS}/../src ${realsense2_INCLUDE_DIRS} )
When I set InferenceEngine_INCLUDE_DIRS manually , this proces passed sucessfully.

The InferenceEngine_INCLUDE_DIRS is not defined in ros2_openvino project, it is defined in openvino (/dldt-2019/inference-engine/src/CMakeLists.txt) and this variable is cancelled in the latest version.
So, what about using the relative path of InferenceEngine_DIR instead ofInferenceEngine_INCLUDE_DIRS ?

request from Object analytics integrate with ros2_openvino_toolkit

@LewisLiuPub We would like to adopt ros2_openvino_toolkit to provide object detection information for ros2_object_analytics input resource as second option with NCS. We have several requests for the ros2_openvino_toolkit package while integration, would you please help to verify and support? many thanks.

1. Keep the same frame_id for output topic with input topic. For example Realsense publish topic with "camera_depth_optical_frame" and openvino should also keep the same frame ID to message publish. Applications need to check frame ID to ensure the same message resource.
2. Provide an easy way to launch openvino with customable input topic. Ros2 launch enable remap and argument customize. Would you please reference to https://github.com/intel/ros2_object_analytics/blob/master/object_analytics_node/launch/object_analytics.launch.py
3. Based on #2 above, suggest to set INPUT_TOPIC as generic name and remap to customized name in launcher.py
4. Hardcode sample/param/pipeline_object_oss.yaml is not flexible. Suggest to added interface with "--input" or added more config file such as xx_for_RealSenseCameraTopic.xml and xx_for_StandardCamera.xml for current usage.

About "use_intra_process_comms" can't work

Hi
in the robot_devkit branch, when I run rs_od file, it gives me :

[INFO] [realsense_node-1]: process started with pid [47019]
[realsense_node-1] [INFO] [camera]: Device with serial number 837212070949 was found.
[realsense_node-1] [INFO] [camera]: Create a node for D435i Camera
[realsense_node-1] terminate called after throwing an instance of 'std::invalid_argument'
[realsense_node-1] what(): intraprocess communication allowed only with volatile durability
[ERROR] [realsense_node-1]: process has died [pid 47019, exit code -6, cmd '/home/dongxuan/dev_ws/install/realsense_node/lib/realsense_node/realsense_node --ros-args --params-file /tmp/launch_params_k0e9tlse'].

my ros2 version is eloquent.

Unable to build with the lastest ros2_core.

Operating environment:

• ubuntu18.04
• ros2_core master branch(20190520)
• openvino_R5
• librealsense 2.17.1
• opencv 3.4.2
• DDS fastrtps

ros2_openvino_toolkit

devel branch ff542bd

Log information

[Processing: dynamic_vino_lib]                                  
--- stderr: dynamic_vino_lib                                    
/home/intel/openvino_ws/src/ros2_openvino_toolkit/dynamic_vino_lib/src/inputs/realsense_camera_topic.cpp: In member function ‘virtual bool Input::RealSenseCameraTopic::initialize()’:
/home/intel/openvino_ws/src/ros2_openvino_toolkit/dynamic_vino_lib/src/inputs/realsense_camera_topic.cpp:39:70: error: ‘std::shared_ptr<SubscriptionT> rclcpp::Node::create_subscription(const string&, CallbackT&&, const rmw_qos_profile_t&, rclcpp::callback_group::CallbackGroup::SharedPtr, bool, typename rclcpp::message_memory_strategy::MessageMemoryStrategy<typename rclcpp::subscription_traits::has_message_type<CallbackT>::type, AllocatorT>::SharedPtr, std::shared_ptr<PublisherT>) [with MessageT = sensor_msgs::msg::Image_<std::allocator<void> >; CallbackT = std::_Bind<void (Input::RealSenseCameraTopic::*(Input::RealSenseCameraTopic*, std::_Placeholder<1>))(std::shared_ptr<sensor_msgs::msg::Image_<std::allocator<void> > >)>; Alloc = std::allocator<void>; SubscriptionT = rclcpp::Subscription<sensor_msgs::msg::Image_<std::allocator<void> > >; std::__cxx11::string = std::__cxx11::basic_string<char>; rmw_qos_profile_t = rmw_qos_profile_t; rclcpp::callback_group::CallbackGroup::SharedPtr = std::shared_ptr<rclcpp::callback_group::CallbackGroup>; typename rclcpp::message_memory_strategy::MessageMemoryStrategy<typename rclcpp::subscription_traits::has_message_type<CallbackT>::type, AllocatorT>::SharedPtr = std::shared_ptr<rclcpp::message_memory_strategy::MessageMemoryStrategy<sensor_msgs::msg::Image_<std::allocator<void> >, std::allocator<void> > >]’ is deprecated: use create_subscription(const std::string &, const rclcpp::QoS &, CallbackT, ...) instead [-Werror=deprecated-declarations]
     std::bind(&RealSenseCameraTopic::cb, this, std::placeholders::_1));
                                                                      ^
In file included from /home/intel/ros2_ws/install/rclcpp/include/rclcpp/node.hpp:1205:0,
                 from /home/intel/ros2_ws/install/rclcpp/include/rclcpp/executors/single_threaded_executor.hpp:28,
                 from /home/intel/ros2_ws/install/rclcpp/include/rclcpp/executors.hpp:22,
                 from /home/intel/ros2_ws/install/rclcpp/include/rclcpp/rclcpp.hpp:144,
                 from /home/intel/openvino_ws/src/ros2_openvino_toolkit/dynamic_vino_lib/include/dynamic_vino_lib/inputs/realsense_camera_topic.hpp:23,
                 from /home/intel/openvino_ws/src/ros2_openvino_toolkit/dynamic_vino_lib/src/inputs/realsense_camera_topic.cpp:22:
/home/intel/ros2_ws/install/rclcpp/include/rclcpp/node_impl.hpp:139:1: note: declared here
 Node::create_subscription(
 ^~~~
cc1plus: all warnings being treated as errors
make[2]: *** [CMakeFiles/dynamic_vino_lib.dir/src/inputs/realsense_camera_topic.cpp.o] Error 1
make[2]: *** Waiting for unfinished jobs....
make[1]: *** [CMakeFiles/dynamic_vino_lib.dir/all] Error 2
make: *** [all] Error 2
---
Failed   <<< dynamic_vino_lib	[ Exited with code 2 ]

Installation guide Issue

doc/BINARY_VERSION_README.md
at 4.Building and Installation
after cd /opt/intel/openvino/deployment_tools/inference_engine/samples/
I recommand run ./build_samples.sh instead of manually build the samples
and then:
export CPU_EXTENSION_LIB=$HOME/inference_engine_samples_build/intel64/Release/lib/libcpu_extension.so
export GFLAGS_LIB=$HOME/inference_engine_samples_build/intel64/Release/lib/libgflags_nothreads.a

ros2 foxy release

Some date about ros2 foxy release?

Unknown CMake command "ament_export_targets".

Based on the guidance document BINARY_VERSION_README in ROS2_openvino_Toolkit, setup environment is ROS dashing and OpenVino 2020.3. I now have a question whether Ament compiling tools are required. According to the information, Presenting Ament is a previous generation of Colcon compiling tools. Since I can't configure Ament at the moment, how can I solve this problem:CMake Error at cmakelists. TXT :58 (ament_export_targets): Unknown CMake command "ament_export_targets".

Wrong fps when launching object detection input from RealSenseCameraTopic

When launching object detection input from RealSenseCameraTopic:

ros2 launch dynamic_vino_sample pipeline_object_topic.launch.py

showing the wrong fps(about 400/fps).

change sample pictures' names for better understanding

pictures under folder /data/images:
car.png -->sample_car.png
team.jpg-->sample_faces.jpg

Yolov3 support in Object detection

According to the documentation I can see SSD and Yolov2 models are supported.
Does this library support also higher versions of yolo like v3 and also are there plans to support other popular object detection models like faster R-CNN?

Dependency problem in pipiline_manager

Implementation dependency problem of class CreateFaceDtection and CreateObjectDetection needs to be fixed in pipeline_manager

Preferred version of Ubuntu

The ROS2 version I intend to use is Foxy. Foxy's preferred Ubuntu version is 20.04.

OpenVino 2021.1/2021.2 both support Ubuntu 20.04. Will ros2_openvino_toolkit work with OpenVino 2021.1/2021.2?

wrong fps display during multi-pipeline running

It seems that fps counter is a static variable.

RViz output is designed with hard-code implementation

In the process of camera data inference, it is hard-coded when specific output method invocation by judgying the given output name.
it would be better to design new separate class for RViz output handling process.

face detection service should output a strutured result

I'm not able to run OpenVino toolkit

I always get the following error if I want to install OpenVino ROS2 on Ubuntu 18.04 via environment_setup.sh:
Installing LibRealsense
gpg: can't open '/home/.../code/open_model_zoo/demos/build/C8B3A55A6F3EFCDE' : No such file or directory

If I install without librealsense, I will get the following error if I want to launch the package.

An if I start: ros2 launch dynamic_vino_sample pipeline_people.launch.py

I will get the following error:
Package 'dynamic_vino_sample' not found: "package 'dynamic_vino_sample' not found, searching: ['/opt/ros/dashing']"

build failed when building dynamic_vino_lib

$colcon build --symlink-install

--- stderr: dynamic_vino_lib
CMake Error at CMakeLists.txt:68 (set_target_properties):
set_target_properties called with incorrect number of arguments.

make: *** [cmake_check_build_system] Error 1

Failed <<< dynamic_vino_lib [ Exited with code 2 ]

Suggested OpenVino version ?

Is there any suggested openvino version ?
I download binary openvino from website (openvino_2020.3.194) and trying to use that with ros2_opevino_tooklit.
I got an error wher env CPU_EXTENSION_LIB is not set. But I could not found any libcpu_extension.dylib under the binary.

PS: I use OSX. v10.14.6

double free or corruption (out) with Yolov2 Object detection

I am trying to deploy a custom trained YoloV2 model for the object detection pipeline in the dynamic_vino_lib samples. However I am getting a double free or corruption (out)
Aborted (core dumped) when I launch the object detection node. Can't tell what the problem is because the log doesn't give any clue. Can you help?

[ INFO ] Loading network files
[ INFO ] Batch size is set to  1
[ INFO ] Checking input size
[ INFO ] Checking output size
[ INFO ] Checking Object Detection outputs
[ INFO ] Checking Object Detection outputs ...
[ INFO ] Checking Object Detection output ... Name=output/YoloRegion
[ INFO ] Checking Object Detection num_classes
[ INFO ] Checking Object Detection output ... num_classes=3
[ INFO ] max proposal count is: 110
[ INFO ]  ... Adding one Inference: ObjectDetection
[ INFO ] Updating connections ...
[ INFO ] Checking connection into pipeline:[ObjectDetection(2)<-->RosTopic(3)]
[ INFO ] Adding connection into pipeline:[ObjectDetection<-->RosTopic]
[ INFO ] Checking connection into pipeline:[RealSenseCameraTopic(1)<-->ObjectDetection(2)]
[ INFO ] Adding connection into pipeline:[RealSenseCameraTopic<-->ObjectDetection]
[ INFO ] One Pipeline Created!
 --> RealSenseCameraTopic
ObjectDetection --> RosTopic
RealSenseCameraTopic --> ObjectDetection
double free or corruption (out)
Aborted (core dumped)

download openvino_toolkit fail when run environment_setup_binary.sh

$ wget -c http://registrationcenter-download.intel.com/akdlm/irc_nas/15512/l_openvino_toolkit_p_2019.1.144.tgz
--2019-09-11 13:47:33-- http://registrationcenter-download.intel.com/akdlm/irc_nas/15512/l_openvino_toolkit_p_2019.1.144.tgz
Resolving child-prc.intel.com (child-prc.intel.com)... 10.239.4.101
Connecting to child-prc.intel.com (child-prc.intel.com)|10.239.4.101|:913... connected.
Proxy request sent, awaiting response... 503 Service Unavailable
2019-09-11 13:47:33 ERROR 503: Service Unavailable.

Error in Step 3. Building and Installation

Hello,
I have successfully completed all steps to 3.
when I now try to execute the CMake command in step 3 I get the following error.

CMake Error: The source directory "/opt/intel/openvino_2021/deployment_tools/inference_engine/samples" does not appear to contain CMakeLists.txt.

Did something go wrong with the installation?

Thanks in advance.

not found setprecision while compile dynamic_vino_lib

Openvino build error while developing https://github.com/intel/robot_devkit.git (intel/robot_devkit#77)
It looks like the the dependence relations is changes and the file "image_window_output.cpp" miss the include

--- stderr: dynamic_vino_lib
/home/rdktest/robot_devkit/rdk_ws/perception_ws/src/intel/ros2_openvino_toolkit/dynamic_vino_lib/src/outputs/image_window_output.cpp: In member function ‘virtual void Outputs::ImageWindowOutput::accept(const std::vector<dynamic_vino_lib::ObjectSegmentationResult>&)’:
/home/rdktest/robot_devkit/rdk_ws/perception_ws/src/intel/ros2_openvino_toolkit/dynamic_vino_lib/src/outputs/image_window_output.cpp:178:44: error: ‘setprecision’ is not a member of ‘std’
ostream << "[" << std::fixed << std::setprecision(3) << fd_conf << "]";
^~~~~~~~~~~~
/home/rdktest/robot_devkit/rdk_ws/perception_ws/src/intel/ros2_openvino_toolkit/dynamic_vino_lib/src/outputs/image_window_output.cpp:178:44: note: suggested alternative: ‘streamsize’
ostream << "[" << std::fixed << std::setprecision(3) << fd_conf << "]";
^~~~~~~~~~~~
streamsize
/home/rdktest/robot_devkit/rdk_ws/perception_ws/src/intel/ros2_openvino_toolkit/dynamic_vino_lib/src/outputs/image_window_output.cpp: In member function ‘virtual void Outputs::ImageWindowOutput::accept(const std::vector<dynamic_vino_lib::FaceDetectionResult>&)’:
/home/rdktest/robot_devkit/rdk_ws/perception_ws/src/intel/ros2_openvino_toolkit/dynamic_vino_lib/src/outputs/image_window_output.cpp:197:44: error: ‘setprecision’ is not a member of ‘std’
ostream << "[" << std::fixed << std::setprecision(3) << fd_conf << "]";
^~~~~~~~~~~~
/home/rdktest/robot_devkit/rdk_ws/perception_ws/src/intel/ros2_openvino_toolkit/dynamic_vino_lib/src/outputs/image_window_output.cpp:197:44: note: suggested alternative: ‘streamsize’
ostream << "[" << std::fixed << std::setprecision(3) << fd_conf << "]";
^~~~~~~~~~~~
streamsize
/home/rdktest/robot_devkit/rdk_ws/perception_ws/src/intel/ros2_openvino_toolkit/dynamic_vino_lib/src/outputs/image_window_output.cpp: In member function ‘virtual void Outputs::ImageWindowOutput::accept(const std::vector<dynamic_vino_lib::ObjectDetectionResult>&)’:
/home/rdktest/robot_devkit/rdk_ws/perception_ws/src/intel/ros2_openvino_toolkit/dynamic_vino_lib/src/outputs/image_window_output.cpp:213:44: error: ‘setprecision’ is not a member of ‘std’
ostream << "[" << std::fixed << std::setprecision(3) << fd_conf << "]";
^~~~~~~~~~~~
/home/rdktest/robot_devkit/rdk_ws/perception_ws/src/intel/ros2_openvino_toolkit/dynamic_vino_lib/src/outputs/image_window_output.cpp:213:44: note: suggested alternative: ‘streamsize’
ostream << "[" << std::fixed << std::setprecision(3) << fd_conf << "]";
^~~~~~~~~~~~
streamsize
/home/rdktest/robot_devkit/rdk_ws/perception_ws/src/intel/ros2_openvino_toolkit/dynamic_vino_lib/src/outputs/image_window_output.cpp: In member function ‘virtual void Outputs::ImageWindowOutput::accept(const std::vector<dynamic_vino_lib::AgeGenderResult>&)’:
/home/rdktest/robot_devkit/rdk_ws/perception_ws/src/intel/ros2_openvino_toolkit/dynamic_vino_lib/src/outputs/image_window_output.cpp:242:43: error: ‘setprecision’ is not a member of ‘std’
ostream << "[Y" << std::fixed << std::setprecision(0) << results[i].getAge() << "]";
^~~~~~~~~~~~
/home/rdktest/robot_devkit/rdk_ws/perception_ws/src/intel/ros2_openvino_toolkit/dynamic_vino_lib/src/outputs/image_window_output.cpp:242:43: note: suggested alternative: ‘streamsize’
ostream << "[Y" << std::fixed << std::setprecision(0) << results[i].getAge() << "]";
^~~~~~~~~~~~
streamsize
make[2]: *** [CMakeFiles/dynamic_vino_lib.dir/src/outputs/image_window_output.cpp.o] Error 1
make[2]: *** Waiting for unfinished jobs....
make[1]: *** [CMakeFiles/dynamic_vino_lib.dir/all] Error 2
make: *** [all] Error 2

---

Failed <<< dynamic_vino_lib [ Exited with code 2 ]

Build error on 0.5.0_R5

When building with colcon build --symlink-install
There will be errors preventing a success build:

--- stderr: dynamic_vino_lib                                 
/home/ros/ros2_ws/src/ros2_openvino_toolkit/dynamic_vino_lib/src/inputs/standard_camera.cpp: In member function ‘virtual bool Input::StandardCamera::initialize()’:
/home/ros/ros2_ws/src/ros2_openvino_toolkit/dynamic_vino_lib/src/inputs/standard_camera.cpp:25:28: error: ‘CV_CAP_PROP_FRAME_WIDTH’ was not declared in this scope
   setWidth((size_t)cap.get(CV_CAP_PROP_FRAME_WIDTH));
                            ^~~~~~~~~~~~~~~~~~~~~~~
/home/ros/ros2_ws/src/ros2_openvino_toolkit/dynamic_vino_lib/src/inputs/standard_camera.cpp:26:29: error: ‘CV_CAP_PROP_FRAME_HEIGHT’ was not declared in this scope
   setHeight((size_t)cap.get(CV_CAP_PROP_FRAME_HEIGHT));
                             ^~~~~~~~~~~~~~~~~~~~~~~~
/home/ros/ros2_ws/src/ros2_openvino_toolkit/dynamic_vino_lib/src/inputs/standard_camera.cpp: In member function ‘virtual bool Input::StandardCamera::initialize(int)’:
/home/ros/ros2_ws/src/ros2_openvino_toolkit/dynamic_vino_lib/src/inputs/standard_camera.cpp:33:28: error: ‘CV_CAP_PROP_FRAME_WIDTH’ was not declared in this scope
   setWidth((size_t)cap.get(CV_CAP_PROP_FRAME_WIDTH));
                            ^~~~~~~~~~~~~~~~~~~~~~~
/home/ros/ros2_ws/src/ros2_openvino_toolkit/dynamic_vino_lib/src/inputs/standard_camera.cpp:34:29: error: ‘CV_CAP_PROP_FRAME_HEIGHT’ was not declared in this scope
   setHeight((size_t)cap.get(CV_CAP_PROP_FRAME_HEIGHT));
                             ^~~~~~~~~~~~~~~~~~~~~~~~
/home/ros/ros2_ws/src/ros2_openvino_toolkit/dynamic_vino_lib/src/inputs/standard_camera.cpp: In member function ‘virtual bool Input::StandardCamera::initialize(size_t, size_t)’:
/home/ros/ros2_ws/src/ros2_openvino_toolkit/dynamic_vino_lib/src/inputs/standard_camera.cpp:44:13: error: ‘CV_CAP_PROP_FRAME_WIDTH’ was not declared in this scope
     cap.set(CV_CAP_PROP_FRAME_WIDTH, width);
             ^~~~~~~~~~~~~~~~~~~~~~~
/home/ros/ros2_ws/src/ros2_openvino_toolkit/dynamic_vino_lib/src/inputs/standard_camera.cpp:45:13: error: ‘CV_CAP_PROP_FRAME_HEIGHT’ was not declared in this scope
     cap.set(CV_CAP_PROP_FRAME_HEIGHT, height);
             ^~~~~~~~~~~~~~~~~~~~~~~~
make[2]: *** [CMakeFiles/dynamic_vino_lib.dir/src/inputs/standard_camera.cpp.o] Error 1
make[2]: *** Waiting for unfinished jobs....
/home/ros/ros2_ws/src/ros2_openvino_toolkit/dynamic_vino_lib/src/inputs/video_input.cpp: In member function ‘virtual bool Input::Video::initialize()’:
/home/ros/ros2_ws/src/ros2_openvino_toolkit/dynamic_vino_lib/src/inputs/video_input.cpp:33:28: error: ‘CV_CAP_PROP_FRAME_WIDTH’ was not declared in this scope
   setWidth((size_t)cap.get(CV_CAP_PROP_FRAME_WIDTH));
                            ^~~~~~~~~~~~~~~~~~~~~~~
/home/ros/ros2_ws/src/ros2_openvino_toolkit/dynamic_vino_lib/src/inputs/video_input.cpp:34:29: error: ‘CV_CAP_PROP_FRAME_HEIGHT’ was not declared in this scope
   setHeight((size_t)cap.get(CV_CAP_PROP_FRAME_HEIGHT));
                             ^~~~~~~~~~~~~~~~~~~~~~~~
/home/ros/ros2_ws/src/ros2_openvino_toolkit/dynamic_vino_lib/src/inputs/video_input.cpp: In member function ‘virtual bool Input::Video::initialize(size_t, size_t)’:
/home/ros/ros2_ws/src/ros2_openvino_toolkit/dynamic_vino_lib/src/inputs/video_input.cpp:44:13: error: ‘CV_CAP_PROP_FRAME_WIDTH’ was not declared in this scope
     cap.set(CV_CAP_PROP_FRAME_WIDTH, width);
             ^~~~~~~~~~~~~~~~~~~~~~~
/home/ros/ros2_ws/src/ros2_openvino_toolkit/dynamic_vino_lib/src/inputs/video_input.cpp:45:13: error: ‘CV_CAP_PROP_FRAME_HEIGHT’ was not declared in this scope
     cap.set(CV_CAP_PROP_FRAME_HEIGHT, height);
             ^~~~~~~~~~~~~~~~~~~~~~~~
make[2]: *** [CMakeFiles/dynamic_vino_lib.dir/src/inputs/video_input.cpp.o] Error 1
make[1]: *** [CMakeFiles/dynamic_vino_lib.dir/all] Error 2
make: *** [all] Error 2
---
Failed   <<< dynamic_vino_lib	[ Exited with code 2 ]

Summary: 10 packages finished [29.3s]
  1 package failed: dynamic_vino_lib
  1 package had stderr output: dynamic_vino_lib
  1 package not processed

and

  /home/ros/ros2_ws/src/ros2_openvino_toolkit/sample/src/image_object_client.cpp: In function ‘int main(int, char**)’:
/home/ros/ros2_ws/src/ros2_openvino_toolkit/sample/src/image_object_client.cpp:83:28: error: ‘cvPoint’ was not declared in this scope
       cv::rectangle(image, cvPoint(xmin, ymin), cvPoint(xmax, ymin + 20), cv::Scalar(0, 255, 0),
                            ^~~~~~~

people_msgs build failed when using opensplice.

The error log is shown below:

[Processing: object_analytics_msgs, people_msgs]                                                                  
--- stderr: people_msgs                                                                                            
In file included from /home/intel/robot_sdk/sdk_ws/modules_ws/build/people_msgs/rosidl_typesupport_opensplice_cpp/people_msgs/srv/dds_opensplice/AgeGender_.h:15:0,
                 from /home/intel/robot_sdk/sdk_ws/modules_ws/build/people_msgs/rosidl_typesupport_opensplice_cpp/people_msgs/srv/dds_opensplice/ccpp_AgeGender_.h:4,
                 from /home/intel/robot_sdk/sdk_ws/modules_ws/build/people_msgs/rosidl_typesupport_opensplice_cpp/people_msgs/srv/age_gender__rosidl_typesupport_opensplice_cpp.hpp:14,
                 from /home/intel/robot_sdk/sdk_ws/modules_ws/build/people_msgs/rosidl_typesupport_opensplice_cpp/people_msgs/srv/dds_opensplice/age_gender__type_support.cpp:19:
/home/intel/robot_sdk/sdk_ws/modules_ws/build/people_msgs/rosidl_typesupport_opensplice_cpp/people_msgs/msg/dds_opensplice/AgeGenderStamped_.h:30:79: error: type/value mismatch at argument 1 in template parameter list for ‘template<class T, class X> class DDS_DCPSUVLSeq’
                typedef DDS_DCPSUVLSeq < AgeGender_, struct _objects__seq_uniq_> _objects__seq;
                                                                               ^
/home/intel/robot_sdk/sdk_ws/modules_ws/build/people_msgs/rosidl_typesupport_opensplice_cpp/people_msgs/msg/dds_opensplice/AgeGenderStamped_.h:30:79: note:   expected a type, got ‘AgeGender_’
In file included from /home/intel/robot_sdk/sdk_ws/modules_ws/build/people_msgs/rosidl_typesupport_opensplice_cpp/people_msgs/srv/dds_opensplice/Emotion_.h:15:0,
                 from /home/intel/robot_sdk/sdk_ws/modules_ws/build/people_msgs/rosidl_typesupport_opensplice_cpp/people_msgs/srv/dds_opensplice/ccpp_Emotion_.h:4,
                 from /home/intel/robot_sdk/sdk_ws/modules_ws/build/people_msgs/rosidl_typesupport_opensplice_cpp/people_msgs/srv/emotion__rosidl_typesupport_opensplice_cpp.hpp:14,
                 from /home/intel/robot_sdk/sdk_ws/modules_ws/build/people_msgs/rosidl_typesupport_opensplice_cpp/people_msgs/srv/dds_opensplice/emotion__type_support.cpp:19:
/home/intel/robot_sdk/sdk_ws/modules_ws/build/people_msgs/rosidl_typesupport_opensplice_cpp/people_msgs/msg/dds_opensplice/EmotionsStamped_.h:30:78: error: type/value mismatch at argument 1 in template parameter list for ‘template<class T, class X> class DDS_DCPSUVLSeq’
                typedef DDS_DCPSUVLSeq < Emotion_, struct _emotions__seq_uniq_> _emotions__seq;
                                                                              ^
/home/intel/robot_sdk/sdk_ws/modules_ws/build/people_msgs/rosidl_typesupport_opensplice_cpp/people_msgs/msg/dds_opensplice/EmotionsStamped_.h:30:78: note:   expected a type, got ‘Emotion_’
In file included from /home/intel/robot_sdk/sdk_ws/modules_ws/build/people_msgs/rosidl_typesupport_opensplice_cpp/people_msgs/srv/dds_opensplice/HeadPose_.h:15:0,
                 from /home/intel/robot_sdk/sdk_ws/modules_ws/build/people_msgs/rosidl_typesupport_opensplice_cpp/people_msgs/srv/dds_opensplice/ccpp_HeadPose_.h:4,
                 from /home/intel/robot_sdk/sdk_ws/modules_ws/build/people_msgs/rosidl_typesupport_opensplice_cpp/people_msgs/srv/head_pose__rosidl_typesupport_opensplice_cpp.hpp:14,
                 from /home/intel/robot_sdk/sdk_ws/modules_ws/build/people_msgs/rosidl_typesupport_opensplice_cpp/people_msgs/srv/dds_opensplice/head_pose__type_support.cpp:19:
/home/intel/robot_sdk/sdk_ws/modules_ws/build/people_msgs/rosidl_typesupport_opensplice_cpp/people_msgs/msg/dds_opensplice/HeadPoseStamped_.h:30:80: error: type/value mismatch at argument 1 in template parameter list for ‘template<class T, class X> class DDS_DCPSUFLSeq’
                typedef DDS_DCPSUFLSeq < HeadPose_, struct _headposes__seq_uniq_> _headposes__seq;
                                                                                ^
/home/intel/robot_sdk/sdk_ws/modules_ws/build/people_msgs/rosidl_typesupport_opensplice_cpp/people_msgs/msg/dds_opensplice/HeadPoseStamped_.h:30:80: note:   expected a type, got ‘HeadPose_’
make[2]: *** [CMakeFiles/people_msgs__rosidl_typesupport_opensplice_cpp.dir/rosidl_typesupport_opensplice_cpp/people_msgs/srv/dds_opensplice/age_gender__type_support.cpp.o] Error 1
make[2]: *** Waiting for unfinished jobs....
make[2]: *** [CMakeFiles/people_msgs__rosidl_typesupport_opensplice_cpp.dir/rosidl_typesupport_opensplice_cpp/people_msgs/srv/dds_opensplice/emotion__type_support.cpp.o] Error 1
make[2]: *** [CMakeFiles/people_msgs__rosidl_typesupport_opensplice_cpp.dir/rosidl_typesupport_opensplice_cpp/people_msgs/srv/dds_opensplice/head_pose__type_support.cpp.o] Error 1
make[1]: *** [CMakeFiles/people_msgs__rosidl_typesupport_opensplice_cpp.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....
make: *** [all] Error 2
---
Failed   <<< people_msgs	[ Exited with code 2 ]

New development features in Y2021?

Hi, ROSers,

I am now planning the new features and development plans for ROS/ROS2 OpenVINO Toolkit projects.
In order to better fit the feature requirements with the limited work load, I would like to gather requirements from the developers, real users or field experts as the first step, then filter them in a feasible development priorities and schedules.

Here are some thoughts about migration/development activities, wish anyone interested can share yours:

• Support latest ROS 2 versions and OpenVINO versions
• Support more popular models and inferences, to better fix robotics use scenarios
• GUI-based AI Inference Pipeline Configuration tool, to ease the OpenVINO Toolkit's use and deployment.
• Docker image support

Please feel free to share your thoughts, what features do you want this project have?

dynamic_vino_sample use case

with ros2 launch dynamic_vino_sample pipeline_segmentation.launch.py

• the error information:

[ ERROR ] Error loading xmlfile: /opt/openvino_toolkit/models/segmentation/output/FP16/frozen_inference_graph.xml, File was not found at line: 1 pos: 0

the config file ros2_openvino_toolkit/sample/param/pipeline_segmentation.yaml

Pipelines:
- name: segmentation
  inputs: [RealSenseCameraTopic]
  infers:
    - name: ObjectSegmentation
      model: /opt/openvino_toolkit/models/segmentation/output/FP16/frozen_inference_graph.xml
      engine: "HETERO:CPU,GPU,MYRIAD"
      label: to/be/set/xxx.labels
      batch: 1
      confidence_threshold: 0.5
  outputs: [ImageWindow, RosTopic, RViz]
  connects:
    - left: RealSenseCameraTopic
      right: [ObjectSegmentation]
    - left: ObjectSegmentation
      right: [ImageWindow]
    - left: ObjectSegmentation
      right: [RosTopic]
    - left: ObjectSegmentation
      right: [RViz]

OpenvinoCommon:

Installed opencv 3.4 but also required opencv 3.3

follow the readme to install opencv3.4, but while launch dynamic_vino_sample, it will still require 3.3. that's should be a issue.

$ ros2 launch dynamic_vino_sample pipeline_people_oss.launch.py
[INFO] [launch]: process[pipeline_with_params-1]: started with pid [11246]
[INFO] [launch]: process[rviz2-2]: started with pid [11247]
/data/proj/oa/ros2_ws/install/dynamic_vino_sample/lib/dynamic_vino_sample/pipeline_with_params: error while loading shared libraries: libopencv_imgcodecs.so.3.3: cannot open shared object file: No such file or directory

No lauching steps for realsense camera

In the Section of Launching Demo, some steps about realsense camera are necessary when input method "RealsenseCameraTopic" is chosen.

error in environment_setup_binary.sh line 79 'http_proxy: unbound variable' on clean Ubuntu 18.04

Hi I'm getting:

ros2_openvino_toolkit/script$ ./environment_setup_binary.sh
Please Enter Your Password:
Begin Environment Setup
Set CLEAN to 1
Set ROS2_SRC to 1
Set OPENVINO to 1
Set OPENCL to 1
Set OPENCV to 1
Set LIBREALSENSE to 0
Set OTHER_DEPENDENCY to 1

...

./environment_setup_binary.sh: line 79: http_proxy: unbound variable

on a fresh ubuntu 18.04 installation
changing line 79:

echo $ROOT_PASSWD | sudo -S apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --keyserver-options http-proxy="$http_proxy" --recv-key F42ED6FBAB17C654

to:

echo $ROOT_PASSWD | sudo -S apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-key F42ED6FBAB17C654

fixed this issue for me.

Incorrect recognition with Text recognition model

Hi,

I`m implementing a text recognition pipeline following this demo (https://github.com/opencv/open_model_zoo/tree/master/demos/text_detection_demo).

I'm using the following pipeline:
TextDetection
TextRecognition

My first fetchResults() method implemented was:

bool dynamic_vino_lib::TextRecognition::fetchResults() {

  bool can_fetch = dynamic_vino_lib::BaseInference::fetchResults();
  if (!can_fetch) {return false;}
  bool found_result = false;

  InferenceEngine::InferRequest::Ptr request = getEngine()->getRequest();
  std::string output = valid_model_->getOutputName();
  InferenceEngine::BlobMap blobs;
  blobs[output] = request->GetBlob(output);

  std::string res = "";
  const char kPadSymbol = '#';
  double conf = 1.0;
  std::string kAlphabet = "0123456789abcdefghijklmnopqrstuvwxyz";
  kAlphabet.push_back(kPadSymbol);

  auto output_shape = blobs.begin()->second->getTensorDesc().getDims();

  for (int i = 0; i < getResultsLength(); i++) {
    
    auto ouput_data_pointer = blobs[output]->buffer().as<float *>();
    std::vector<float> output_data(ouput_data_pointer, ouput_data_pointer + output_shape[0] * output_shape[2]);
    res = CTCGreedyDecoder(output_data, kAlphabet, kPadSymbol, &conf);
    results_[i].result_value = res;

  }

  found_result = true;
  if (!found_result) results_.clear();
  return true;
}

Using this method the network output was:

So my pointer to output_data does not changed as expected. After that, I changed my fetchResults() method using the same structure of head_pose_detection example:

bool dynamic_vino_lib::TextRecognition::fetchResults() {

  bool can_fetch = dynamic_vino_lib::BaseInference::fetchResults();
  if (!can_fetch) {return false;}
  bool found_result = false;

  InferenceEngine::InferRequest::Ptr request = getEngine()->getRequest();

  std::string output = valid_model_->getOutputName();
 
  InferenceEngine::BlobMap blobs;

  blobs[output] = request->GetBlob(output);

  std::string res = "";
  const char kPadSymbol = '#';
  double conf = 1.0;
  std::string kAlphabet = "0123456789abcdefghijklmnopqrstuvwxyz";
  kAlphabet.push_back(kPadSymbol);

  auto output_shape = blobs.begin()->second->getTensorDesc().getDims();


  for (int i = 0; i < getResultsLength(); i++) {
   
    auto ouput_data_pointer = &blobs[output]->buffer().as<float *>()[i];

    std::vector<float> output_data(ouput_data_pointer, ouput_data_pointer + output_shape[0] * output_shape[2]);

    res = CTCGreedyDecoder(output_data, kAlphabet, kPadSymbol, &conf);

    results_[i].result_value = res;

    slog::info << "Result: " << res  << slog::endl;

    output_data.clear();

    ouput_data_pointer = 0;

    res = "";

  }

  found_result = true;

  if (!found_result) results_.clear();
  return true;
}

And the result:

My first pointer is correct but the second not. I tried to understand why but I'm not able. How can I correctly recognize the second text?

Thanks in advance.

Need to update rviz_yaml_cpp_vendor to yaml_cpp_vendor

Due to the update of ros2 code , we need to update rviz_yaml_cpp_vendor to yaml_cpp_vendor, otherwise, the build will fail

--- stderr: vino_param_lib CMake Error at CMakeLists.txt:19 (find_package): By not providing "Findrviz_yaml_cpp_vendor.cmake" in CMAKE_MODULE_PATH this project has asked CMake to find a package configuration file provided by "rviz_yaml_cpp_vendor", but CMake did not find one.

Launch object detection with custom .xml model

System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow): Yes
OS Platform and Distribution: Ubuntu 18.04
ROS2 distribution: Eloquent
TensorFlow installed from (source or binary): binary
TensorFlow version: 1.14.0
Python version: 3.6.9
ros2_openvino_toolkit branch: Eloquent
opencv/dldt branch: 2020

I have edited the pipeline_object_topic.yaml file to take in a customized model made with tensorflow 1.14. The model is converted to .xml file by using the mo_tf.py script (https://github.com/opencv/dldt/blob/2020/model-optimizer/mo_tf.py) by:

sudo python3 mo_tf.py --input_model=/home/charlotte/MyDataset/test/model/model.pb --tensorflow_use_custom_operations_config /home/charlotte/opencv/dldt/model-optimizer/extensions/front/tf/ssd_support_api_v1.14.json --tensorflow_object_detection_api_pipeline_config /home/charlotte/MyDataset/test/model/pipeline.config --output_dir /home/charlotte/MyDataset

This works well, but I get the following error when launching ros2 launch dynamic_vino_sample pipeline_object_topic.launch.py:
[pipeline_with_params-1] [ ERROR ] Error reading network: cannot parse future versions: 10

I tried changing the IR_version to generate a xml file with IR_version 5, by changing the returning version in this file: https://github.com/opencv/dldt/blob/2020/model-optimizer/mo/pipeline/common.py, which results in a the following header in my .xml file:

When I try to launch the object detection node with this model, I get the following error:

charlotte@charlotte-MacBookAir:~/opencv/dldt/model-optimizer$ ros2 launch dynamic_vino_sample pipeline_object_topic.launch.py
[INFO] [launch]: All log files can be found below /home/charlotte/.ros/log/2020-03-27-16-58-28-573633-charlotte-MacBookAir-30266
[INFO] [launch]: Default logging verbosity is set to INFO
[INFO] [pipeline_with_params-1]: process started with pid [30280]
[pipeline_with_params-1] InferenceEngine: 0x7f34e2b3b060
[pipeline_with_params-1] [ INFO ] Config File Path =/home/charlotte/ros2_overlay_ws/install/dynamic_vino_sample/share/dynamic_vino_sample/param/pipeline_object_topic.yaml
[pipeline_with_params-1] [ INFO ] Pipeline size: 1
[pipeline_with_params-1] [ INFO ] Inferences size: 1
[pipeline_with_params-1] [ WARNING ] invalid node; this may result from using a map iterator as a sequence iterator, or vice-versa
[pipeline_with_params-1] [ INFO ] Inference Params:name=ObjectDetection
[pipeline_with_params-1] [ WARNING ] invalid node; this may result from using a map iterator as a sequence iterator, or vice-versa
[pipeline_with_params-1] [ INFO ] Pipeline Params:name=object
[pipeline_with_params-1] [ WARNING ] invalid node; this may result from using a map iterator as a sequence iterator, or vice-versa
[pipeline_with_params-1] [ WARNING ] invalid node; this may result from using a map iterator as a sequence iterator, or vice-versa
[pipeline_with_params-1] [ WARNING ] invalid node; this may result from using a map iterator as a sequence iterator, or vice-versa
[pipeline_with_params-1] [ WARNING ] invalid node; this may result from using a map iterator as a sequence iterator, or vice-versa
[pipeline_with_params-1] [ INFO ] --------parameters DUMP---------------------
[pipeline_with_params-1] [ INFO ] Pipeline: object
[pipeline_with_params-1] [ INFO ] Inputs: RealSenseCameraTopic,
[pipeline_with_params-1] [ INFO ] Outputs: ImageWindow, RosTopic, RViz,
[pipeline_with_params-1] [ INFO ] Inferences:
[pipeline_with_params-1] [ INFO ] Name: ObjectDetection
[pipeline_with_params-1] [ INFO ] Model: /home/charlotte/MyDataset/test/model/model.xml
[pipeline_with_params-1] [ INFO ] Engine: CPU
[pipeline_with_params-1] [ INFO ] Label: to/be/set/xxx.labels
[pipeline_with_params-1] [ INFO ] Batch: 1
[pipeline_with_params-1] [ INFO ] Confidence_threshold: 0.5
[pipeline_with_params-1] [ INFO ] Enable_roi_constraint: 1
[pipeline_with_params-1] [ INFO ] Connections:
[pipeline_with_params-1] [ INFO ] ObjectDetection->ImageWindow
[pipeline_with_params-1] [ INFO ] ObjectDetection->RosTopic
[pipeline_with_params-1] [ INFO ] ObjectDetection->RViz
[pipeline_with_params-1] [ INFO ] RealSenseCameraTopic->ObjectDetection
[pipeline_with_params-1] [ INFO ] Common:
[pipeline_with_params-1] [ INFO ] camera_topic:
[pipeline_with_params-1] [ INFO ] custom_cpu_library:
[pipeline_with_params-1] [ INFO ] custom_cldnn_library:
[pipeline_with_params-1] [ INFO ] enable_performance_count: 0
[pipeline_with_params-1] [ INFO ] Parsing InputDvice: RealSenseCameraTopic
[pipeline_with_params-1] [ INFO ] ... Adding one Input device: RealSenseCameraTopic
[pipeline_with_params-1] [ INFO ] Adding Input Device into Pipeline: RealSenseCameraTopic
[pipeline_with_params-1] [ INFO ] Adding connection into pipeline:[<-->RealSenseCameraTopic]
[pipeline_with_params-1] [ INFO ] Parsing Output: ImageWindow
[pipeline_with_params-1] [ INFO ] ... Adding one Output: ImageWindow
[pipeline_with_params-1] [ INFO ] Parsing Output: RosTopic
[pipeline_with_params-1] [ INFO ] ... Adding one Output: RosTopic
[pipeline_with_params-1] [ INFO ] Parsing Output: RViz
[pipeline_with_params-1] [ INFO ] ... Adding one Output: RViz
[pipeline_with_params-1] [ INFO ] Parsing Inference: ObjectDetection
[pipeline_with_params-1] [ INFO ] Loading network files
[pipeline_with_params-1] [ ERROR ] Error reading network: Error of validate layer: BoxPredictor_0/BoxEncodingPredictor/Conv2D with type: Convolution. No such parameter name 'output' for layer BoxPredictor_0/BoxEncodingPredictor/Conv2D
[ERROR] [pipeline_with_params-1]: process has died [pid 30280, exit code 254, cmd '/home/charlotte/ros2_overlay_ws/install/dynamic_vino_sample/lib/dynamic_vino_sample/pipeline_with_params -config /home/charlotte/ros2_overlay_ws/install/dynamic_vino_sample/share/dynamic_vino_sample/param/pipeline_object_topic.yaml --ros-args -r /openvino_toolkit/object/detected_objects:=/ros2_openvino_toolkit/detected_objects -r /openvino_toolkit/object/images:=/ros2_openvino_toolkit/image_rviz'].

I have no problems running ros2 launch dynamic_vino_sample pipeline_object_topic.launch.py with one of the example models, such as the mobilenet-ssd from here: https://github.com/intel/ros2_openvino_toolkit/blob/master/doc/OBJECT_DETECTION.md.

I`ve tried removing the output from the layer concerned and the edge for the output from my model.xml file, but the error remains.
Is the problem that ros2_openvino_toolkit is not compatible with the 2020 openvino distribution?

Attached is my model.xml file:
model.txt

After get openvino R4 open source code, colcon build occurred error

After get openvino R4 open source code, colcon build occurred error:
--- stderr: dynamic_vino_lib In file included from /home/iei/ros2_overlay_ws/src/ros2_openvino_toolkit/dynamic_vino_lib/src/factory.cpp:25:0: /home/iei/ros2_overlay_ws/src/ros2_openvino_toolkit/dynamic_vino_lib/include/dynamic_vino_lib/factory.hpp:34:43: fatal error: mkldnn/mkldnn_extension_ptr.hpp: No such file or directory compilation terminated. make[2]: *** [CMakeFiles/dynamic_vino_lib.dir/src/factory.cpp.o] Error 1 make[2]: *** Waiting for unfinished jobs....

why need link src after colcon build

from the build package Guide, after colcon build, there is one command to link ros2_openvino_toolkit to /opt/openvino_toolkit, don't know what is the purpose, could this soft link be removed?

"sudo ln -sf ~/ros2_overlay_ws/src/ros2_openvino_toolkit /opt/openvino_toolkit/"

when launching face detection, there was no result in /ros2_openvino_toolkit/face_detection topic

when launching face detection,

ros2 launch dynamic_vino_sample pipeline_people_oss.launch.py

there was no result in /ros2_openvino_toolkit/face_detection topic.

ros2 topic echo /ros2_openvino_toolkit/face_detection

the result is in object detection topic:

ros2 topic echo /openvino_toolkit/detected_objects

OpenVINO R4 binary package and opencv v3.4, colcon build occurred error

I want to verify if OpenVINO R4 binary package and opencv v3.4 can work together, so after installing OpenVINO R4 binary package, I manually installed opencv v3.4. colcon build occurred error:

--- stderr: dynamic_vino_sample                                                                                    
CMake Error at /opt/intel/computer_vision_sdk_2018.4.420/deployment_tools/inference_engine/share/InferenceEngineConfig.cmake:180 (add_subdirectory):
  The source directory
    /home/intel/ros2_overlay_ws/install/dynamic_vino_lib/include/extension
  does not contain a CMakeLists.txt file.
Call Stack (most recent call first):
  CMakeLists.txt:56 (find_package)
CMake Error at /opt/intel/computer_vision_sdk_2018.4.420/deployment_tools/inference_engine/share/InferenceEngineConfig.cmake:181 (add_library):
  add_library cannot create ALIAS target "IE::ie_cpu_extension" because
  target "ie_cpu_extension" does not already exist.
Call Stack (most recent call first):
  CMakeLists.txt:56 (find_package)
---
Failed   <<< dynamic_vino_sample	[ Exited with code 1 ]

Instructions are in wrong order?

Instructions have us installing librealsense first (which modifies the uvcvideo.ko driver of the installed kernel) then has installing the OpenVINO second (which asks us to upgrade to a 4.14 kernel). Doesn't this break/revert the patched librealsense uvcvideo.ko driver?

[robot_devkit]Is there a better way to deal with the situation that label is null?

When I compose multiple nodes into a container process through a launch file, "label:null" in the yaml file will cause the issue as follows:

Future exception was never retrieved
future: <Future finished exception=TypeError('Unexpected type for parameter value None',)>
Traceback (most recent call last):
File "/usr/lib/python3.6/concurrent/futures/thread.py", line 56, in run
result = self.fn(*self.args, **self.kwargs)
File "/home/intel/workspace/ros2_ws/build/launch_ros/launch_ros/actions/load_composable_nodes.py", line 151, in _load_in_sequence
self._load_node(next_composable_node_description, context)
File "/home/intel/workspace/ros2_ws/build/launch_ros/launch_ros/actions/load_composable_nodes.py", line 114, in _load_node
context, composable_node_description.parameters
File "/home/intel/workspace/ros2_ws/build/launch_ros/launch_ros/utilities/to_parameters_list.py", line 49, in to_parameters_list
context, normalize_parameter_dict(yaml.load(f))
File "/home/intel/workspace/ros2_ws/build/launch_ros/launch_ros/utilities/normalize_parameters.py", line 158, in normalize_parameter_dict
raise TypeError('Unexpected type for parameter value {}'.format(repr(value)))

request: ROS1 wrapper

Why is it only available for ROS2 and not ROS1?

some classes' names donot exactly align with their functionality

for example, emotionDetection would be placed with EmotionRecognition, HeadposeDetection --> HeadposeRecognition.
meanwhile, some names' definition is not suitable, for example, one structure and one class have the same name asPipelineParams which hurts the code readability.

error in environment_setup_binary.sh line 146 on clean Ubuntu 18.04

I'm getting an error while running environment_setup_binary.sh with librealsense=1

ros2_openvino_toolkit/script$ ./environment_setup_binary.sh
Please Enter Your Password:
Begin Environment Setup
Set CLEAN to 1
Set ROS2_SRC to 1
Set OPENVINO to 1
Set OPENCL to 1
Set OPENCV to 1
Set LIBREALSENSE to 1
Set OTHER_DEPENDENCY to 1
...
===================Setting Up LibRealSense...=======================
Install server public key for librealsense
gpg: can't open '/home/stawel/Downloads/C8B3A55A6F3EFCDE': no such file or directory

replacing line:

sudo apt-key add ${CURRENT_DIR}/C8B3A55A6F3EFCDE

with:

echo $ROOT_PASSWD | sudo -S apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-key  C8B3A55A6F3EFCDE

fixed this issue for me.

error dynamic_vino_sample' not found

Hello,
I have tried containerizing the application with Docker. When I use the following command inside the container:

ros2 launch dynamic_vino_sample pipeline_face_reidentification.launch.py

I get the below error message:

Package 'dynamic_vino_sample' not found: "package 'dynamic_vino_sample' not found, searching: ['/opt/ros/dashing']"

Any suggestions what I could change?

Time stamps are inconsistent.

The timestamp of the output results is not consistent with the timestamp of realsense.

ros2_overlay_ws colcon build --symlink-install error

During step 4. Building and Installation
-->Install ROS2_OpenVINO packages
-->Build package
cd ~/ros2_overlay_ws
colcon build --symlink-install

[Processing: cv_bridge, message_filters, object_msgs, realsense_msgs]
--- stderr: cv_bridge
make[2]: *** No rule to make target '/usr/lib/x86_64-linux-gnu/libboost_python3.so', needed by 'src/libcv_bridge.so'. Stop.
make[2]: *** Waiting for unfinished jobs....
make[1]: *** [src/CMakeFiles/cv_bridge.dir/all] Error 2
make: *** [all] Error 2

Failed <<< cv_bridge [ Exited with code 2 ]
Aborted <<< realsense_msgs
Aborted <<< message_filters
Aborted <<< object_msgs

Summary: 0 packages finished [2min 4s]
1 package failed: cv_bridge
3 packages aborted: message_filters object_msgs realsense_msgs
1 package had stderr output: cv_bridge
15 packages not processed

Unexpected removal of user space stuff

I'm very, very surprised that the script environment_setup.sh ACTUALLY REMOVES everything without even a warning. I've paid a very high price of loosing everything in my /ros2_ws, which is my ROS2 equivalent of /catkin_ws (since my ROS2 is installed by apt).
And its behavior of grabing the ROS2 Crystal source code and rebuild it from scratch, while there is a viable apt version which is far quicker, doesn't make too much sense to me.

IMHO, the default install configuration in modules.conf should be set to not remove anything.
Documents suggesting to execute that pprticular bash such as BINARY_VERSION_README.md should add some warning text to it.

set_target_properties called with incorrect number of arguments

• branch: master
• ubuntu 18.04
• ros-dashing

Output:

--- stderr: dynamic_vino_lib                            
CMake Error at CMakeLists.txt:81 (set_target_properties):
  set_target_properties called with incorrect number of arguments.


---
Failed   <<< dynamic_vino_lib   [ Exited with code 1 ]