unity-technologies / ros-tcp-connector

ROS TCP Connector

---

We're currently working on lots of things! Please take a short moment fill out our survey to help us identify what products and packages to build next.

---

Introduction

This repository contains two Unity packages: the ROS TCP Connector, for sending/receiving messages from ROS, and the Visualizations Package, for adding visualizations of incoming and outgoing messages in the Unity scene.

Installation

1. Using Unity 2020.2 or later, open the Package Manager from Window -> Package Manager.

2. In the Package Manager window, find and click the + button in the upper lefthand corner of the window. Select Add package from git URL....

   

3. Enter the git URL for the desired package. Note: you can append a version tag to the end of the git url, like #v0.4.0 or #v0.5.0, to declare a specific package version, or exclude the tag to get the latest from the package's main branch.

   1. For the ROS-TCP-Connector, enter https://github.com/Unity-Technologies/ROS-TCP-Connector.git?path=/com.unity.robotics.ros-tcp-connector.
   2. For Visualizations, enter https://github.com/Unity-Technologies/ROS-TCP-Connector.git?path=/com.unity.robotics.visualizations.

4. Click Add.

To install from a local clone of the repository, see installing a local package in the Unity manual.

Tutorials

This Unity package provides four main features:

• ROSConnection: A component that sets up communication between ROS and Unity. See the Unity Robotics Hub for information and tutorials.

• Message Generation: A tool to generate C# classes for ROS message types.

• Visualizations: A suite of default configurations and APIs to visualize incoming and outgoing information from ROS.

  • See the Nav2 SLAM Example for tutorials on using the Visualizations Package!
  • You can also view the package's Usage Information for more details on using the package in your own project.

• ROSGeometry: A set of extensions that convert geometries between Unity and other coordinate frames.

ROS#

Special thanks to the Siemens ROS# Project Team for driving the ROS-Unity Integration Open Source since 2017.

Community and Feedback

The Unity Robotics projects are open-source and we encourage and welcome contributions. If you wish to contribute, be sure to review our contribution guidelines and code of conduct.

Support

For questions or discussions about Unity Robotics package installations or how to best set up and integrate your robotics projects, please create a new thread on the Unity Robotics forum and make sure to include as much detail as possible.

For feature requests, bugs, or other issues, please file a GitHub issue using the provided templates and the Robotics team will investigate as soon as possible.

For any other questions or feedback, connect directly with the Robotics team at [email protected].

License

Apache License 2.0

ros-tcp-connector's Issues

Support ROS 2 C# client library

Recently, Robotec AI released a ROS 2 C# client library https://github.com/RobotecAI/ros2cs
The main purpose of this library is to enable easy communication between a Unity simulation and ROS 2 code.
They show an example here https://github.com/RobotecAI/ros2-for-unity

Are you considering supporting this library for ROS 2 rather than the TCP connector?
One of the design principles of ROS 2 was indeed the possibility to easily create client library in a variety of different languages, in order to reduce performance overhead and need for custom solutions.

Cant switch to ROS1 in the settings

Describe the bug
When trying the ROS SLAM example and switching form ROS2 to ROS1 in the settings it gets stuck at recompiling and it shows some errors as seen in the screenshot. This project also has to start up in safe mode afterwards

To Reproduce
Steps to reproduce the behavior:

1. Clone the project shows above with submodules
2. Open in unity
3. Switch ROS2 to ROS1
4. See error

Console logs / stack traces
see screenshot

Expected behavior
Changing from ROS2 to ROS1

Screenshots

Environment (please complete the following information, where applicable):
2 Computers:

• Unity Version: [Unity 2020.3.21f1, 2020.3.11f1 ]
• Unity machine OS + version: [MacOS Monterey, Ubuntu 20.04]
• ROS machine OS + version: [Docker]
• ROS–Unity communication: [Docker]
• Branch or version: [v0.6.0 (latest)]

Additional context
/

"CONTRIBUTING.md" contains a dead link

In: CONTRIBUTING.md,
Under Section: Code style
The linked text 'Unity Coding package' redirects to:
https://docs.unity3d.com/Packages/[email protected]/manual/index.html
Which returns "Sorry... that page seems to be missing!"

• Branch 'main' & 'dev' both have this issue.

package.json missing

This commit deleted package.json. Can no longer import git repo into Unity.

Incorrect step size for Image message in ToImageMsg()

Issue and Steps to Reproduce
Adding a camera publisher to unity project from https://github.com/Unity-Technologies/Robotics-Nav2-SLAM-Example using
ImageMsg rosImage = mainCameraTexture.ToImageMsg(header);
causes error while using rqt or image_view node.
ImageView.callback_image() while trying to convert image from 'rgba8' to 'rgb8' an exception was thrown (Image is wrongly formed: step < width * byte_depth * num_channels or 4 != 691 * 1 * 4)

I think it is due to incorrect size here in this line of code, which just sends step=4 irrespective of width of image.

ROS-TCP-Connector/com.unity.robotics.ros-tcp-connector/Runtime/MessageGeneration/MessageExtensions.cs

Line 747 in 7f05510

return new ImageMsg(header, height: (uint)tex.height, width: (uint)tex.width, encoding: encoding, is_bigendian: 0, step: 4, data: data);

Problem can be solved by manually assigning step like
image_step = 4;
img_msg.step = image_step * (uint) frame_width;

I will try to reproduce and fix error for different texture types, And open a PR with correct code of ToImageMsg(). Although I am not sure if this is specific to ros2 or a general bug.

Are there any plans for HL2 support?

Are there any plans to add UWP support so that we can connect HL2 and ROS using this package?

Thanks!

Revert the M-Prefix class renaming

Is your feature request related to a problem? Please describe.
I was using the ROS-TCP-Connector in a project and wrote quite some publisher and subscribers, now I wanted to upgrade to the newest version and the naming of all Messages were changed and I would have to change several files. It also makes the usage inconvenient, if I have an compressed Image and want to change its header, the variable is still named header, but the class is now MHeader. A Prefix to a class also violates all the C# / .Net naming guidelines (which are also mentioned in the Contribution Guidelines).

Describe the solution you'd like
Revert the change and remove the M Class-Prefix.

Retrieve topic message types

I see in the RosConnection script there is a function that retrieves the topic names from the server_endpoint. I was wondering if it would be possible to add a feature/function that also allows me to retrieve topic message types. Similarly, adding a way to get if they are subscribers or publishers.

Website Link Issue

On the Main branch of the ROS-TCP-Connector (https://github.com/Unity-Technologies/ROS-TCP-Connector), in the Tutorials section.

Tutorials
ROSConnection: See the Unity Robotics Hub repository for information and tutorials on how to use this component.

This Unity Robotics Hub has an incorrect link, it links to "https://github.com/Unity-Technologies/Unity-Robotics-Hub/blob/master/tutorials/ros_unity_integration/README.md" but I believe it should link to "https://github.com/Unity-Technologies/Unity-Robotics-Hub/blob/main/tutorials/ros_unity_integration/README.md", as there is no master branch.

Unity cannot load the HudPanel

Describe the bug

Unity cannot load the HUDPanel.cs script because HudPanel inherits from MonoBehaviour but the script is named "HUDPanel.cs". The class name and filename are not exact matches (HudPanel vs. HUDPanel) so it doesn't load.

To Reproduce
Steps to reproduce the behavior:

1. Open any project that has the ROS-TCP-Connector
2. Try to add the HUDPanel.cs script to any GameObject
3. See error

Console logs / stack traces

The script don't inherit a native class that can manage a script.

Expected behavior
The script is added to the GameObject

Screenshots
If applicable, add screenshots or videos to help explain your problem.

Environment (please complete the following information, where applicable):

• Unity Version: Unity 2020.3.2f1
• Unity machine OS + version: Ubuntu 18.04
• ROS machine OS + version: Ubuntu 18.04, ROS Melodic
• ROS–Unity communication: loopback/same machine
• Branch or version: ROS-TCP-Connector v0.6.0

Additional context
Add any other context about the problem here.

Ros message generation menu option missing

Hi, during the https://github.com/Unity-Technologies/Unity-Robotics-Hub/blob/main/tutorials/ros_unity_integration/setup.md tutorial,
after adding the https://github.com/Unity-Technologies/ROS-TCP-Connector.git package from git url, the message generation button in menu toolbar is missing.

We are using the Version 2019.4.17f1 (667c8606c536) Personal

We are on Ubuntu 18.04
Using Ros melodic

In the Unity console we have 2 errors

Library/PackageCache/com.unity.robotics.ros-tcp-connector@457dad63a5/Runtime/ROSGeometry/ROSVector3.cs(202,104): error CS1501: No overload for method 'ToString' takes 2 arguments

Library/PackageCache/com.unity.robotics.ros-tcp-connector@457dad63a5/Runtime/ROSGeometry/ROSQuaternion.cs(145,95): error CS1501: No overload for method 'ToString' takes 2 arguments

There is no error during the catkin_ws compilation

Latest ROS-TCP-Connector package version 0.6.0 throws error

I wanted to upgrade the ROS TCP Connector package from v0.2.0light to the newest version v0.6.0 for the Object-Pose-Estimation Project but now I keep getting this error:

Unity.Robotics.ROSTCPConnector.RosTopicState.SendServiceRequest (Unity.Robotics.ROSTCPConnector.MessageGeneration.Message requestMessage, System.Int32 serviceId) (at Library/PackageCache/com.unity.robotics.ros-tcp-connector@7f055108f4/Runtime/TcpConnector/RosTopicState.cs:234)
Unity.Robotics.ROSTCPConnector.ROSConnection.SendServiceMessage[RESPONSE] (System.String rosServiceName, Unity.Robotics.ROSTCPConnector.MessageGeneration.Message serviceRequest) (at Library/PackageCache/com.unity.robotics.ros-tcp-connector@7f055108f4/Runtime/TcpConnector/ROSConnection.cs:310)
Unity.Robotics.ROSTCPConnector.ROSConnection.SendServiceMessage[RESPONSE] (System.String rosServiceName, Unity.Robotics.ROSTCPConnector.MessageGeneration.Message serviceRequest, System.Action`1[T] callback) (at Library/PackageCache/com.unity.robotics.ros-tcp-connector@7f055108f4/Runtime/TcpConnector/ROSConnection.cs:283)
System.Runtime.CompilerServices.AsyncMethodBuilderCore+<>c.<ThrowAsync>b__7_0 (System.Object state) (at <31c0f51ac5a24a22ba784db24f4ba023>:0)
UnityEngine.UnitySynchronizationContext+WorkRequest.Invoke () (at <ad50157ee00e45cdb3c8bd67012f8804>:0)```

Coordinate Frame Transformations in ENU Don't Match The Standard

Describe the bug

In Unity, to convert a vector from unity space RUF (Right, Up, Front) to ENU (East, North, Up), the following transformation is applied:
public Vector3 ConvertFromRUF(Vector3 v) => new Vector3(v.x, v.z, v.y);
This assumes that in Unity a positive x is East, a positive z is North and a positive y is up.
However, to convert a rotation from Unity space RUF (Right, Up, Forward) to ENU (East, North, Up), the following transformation is applied:
public Quaternion ConvertFromRUF(Quaternion q) => new Quaternion(q.x, q.z, q.y, -q.w);
This however assumes when in Unity, an identity rotation (When the forward vector is positive z (0, 0, 1)) would also be an identity rotation in ENU.
This means that in Unity, when facing z forward or North, the orientation in ENU is also the identity.
However in ENU, when the rotation is identity, forward should be facing East, see:
"In case of land vehicles like cars, tanks etc., which use the ENU-system (East-North-Up) as external reference (World frame), the vehicle's (body's) positive y- or pitch axis always points to its left"

Image from Wikipedia shows the zero point on the yaw to be East, rotating counter-clockwise

• https://en.wikipedia.org/wiki/Axes_conventions#World_reference_frames_for_attitude_description

The problem seems to be that:
Unity geographic coordinates have +x = East, +y = Up and +z = North, but when converting geographic rotations offsets are not taken into account 0 = North when it should be 0 = East (In ENU terms).

We found this discrepancy to be an issue when using Unity to simulate IMU data (converted to ENU before publishing) which would then be fed into the ROS robot_localization package. When doing this, we found that we would be pitching when the object was rolling in Unity and vise versa. The robot_localization link provided indicates that ENU should be used for all IMU data. Switching the conversion to FLU and re-framing our environment such that -x = North, +y = Up and +z = East corrected this discrepancy.

To correctly convert a rotation from Unity space to ENU, either:

1. Unity should be -x = North, +y = Up and +z = East and ENU transformations should be equivalent to FLU transformations. (Note: This would probably break NED rotational transformations)
   OR
2. Unity should apply a yaw offset of 90 degrees when converting from Unity orientations to ENU orientations and vice versa.

Is there something else about the Unity implementation that I am missing?

Some related links

• REP-103 Standard Units of Measure and Coordinate Conventions
• REP-105 Coordinate Frames for Mobile Platforms
• REP-145 Conventions for IMU Sensor Drivers

tf broadcast

I am writing a mobile robot base software and could successfully move it using ROS Publishers and Subscribers. Now I am implementing odometry (following this tutorial) and I realized that ROS-TCP-Connector does not seem to support tf or tf2.

Is there any plan to add support to tf on ROS-TCP-Connector anytime soon? Or can you give me a suggestion on a workaround?

Thanks a lot!

Parameter service

Is your feature request related to a problem? Please describe.
I want to be able to set/get parameters for a Unity ROS node. I think it can be done from scratch using the service API in ROS2, but it would be nice to have an easy interface.

Describe the solution you'd like
A generalized class that allows this to be done easily and arbitrarily for all nodes.

Describe alternatives you've considered
I think its possible to use ROS2 services to manually create this (ROS2 parameters are implemented as services)

Additional context
N/A

Weird behavour when publishing ClockMsg at high frequency

Hi

I'm trying to publish a message of time RosMessageTypes.Rosgraph.ClockMsg every fixed time step in the simulation (time_step=0.01). However, when i print the topic in the terminal a see a weird behavior.

it is sending some time future time stamps, that the simulator hasn't arrived yet. Since I print the values in the unity terminal and the displays values are correct

NotwithStanding, if I use a bigger type of message, i.e RosMessageTypes.Sensor.JointStateMsg and embedded the clock message there. I got nice results. When I tried with message types like HeaderMsg it also present a weird behaviour.

Here is my code

public class ClockSimulation : MonoBehaviour
{
    // Start is called before the first frame update
    Unity.Robotics.ROSTCPConnector.ROSConnection ros;
    public string clock_topic="clock";
    public string clock_real_topic="clock_real";
    void Start()
    {
        ros = Unity.Robotics.ROSTCPConnector.ROSConnection.GetOrCreateInstance();
        ros.RegisterPublisher<RosMessageTypes.Sensor.JointStateMsg>(clock_topic);
        ros.RegisterPublisher<RosMessageTypes.Rosgraph.ClockMsg>(clock_real_topic);
    }

    // Update is called once per frame
    void FixedUpdate()
    {
        var sensor_msg= new RosMessageTypes.Sensor.JointStateMsg();
        sensor_msg.header=new RosMessageTypes.Std.HeaderMsg();
        sensor_msg.header.stamp=ROSCommon.getFixedTimeStamp();
        var clock_msg=new RosMessageTypes.Rosgraph.ClockMsg();
        clock_msg.clock=sensor_msg.header.stamp;
     
        Debug.Log("seconds "+clock_msg.clock.sec+ "nano secs"+clock_msg.clock.nanosec);
       
        ros.Publish(clock_topic,sensor_msg);
        ros.Publish(clock_real_topic,clock_msg);
    }
}

where the function getFixedTimeStamp is:

static public  RosMessageTypes.BuiltinInterfaces.TimeMsg getFixedTimeStamp()
    {
        float time_since_start=Time.fixedTime;
        uint seconds=(uint)Math.Floor(time_since_start);
        uint nano_seconds=(uint)((time_since_start-(float)seconds)*10E9);
        RosMessageTypes.BuiltinInterfaces.TimeMsg time_stamp=new RosMessageTypes.BuiltinInterfaces.TimeMsg(seconds,nano_seconds);
        return time_stamp;
    }

Anyone has an idea of why this is happening? Thank you

Environment :

• Unity Version: Unity 2020.3.21f1
• ROS machine OS + version: Ubuntu 20.04, ROS Noetic
• Branch or version: v0.7.1

packaged to Android

Hi, bros!
I wonder if this communication module can be packaged in Unity project and run on Android platform?

Wrong namespace for `Duration` message from `std_msgs`

Hi,

The Duration message from std_msgs is in the wrong namespace in this package.

In the file MessageGeneration>std_msgs>Duration.cs:

// This should be "namespace RosMessageTypes.Std" like all the other standard messages
namespace Unity.Robotics.ROSTCPConnector.MessageGeneration
{
    public class Duration : Message
    {

cannot convert from 'RosMessageTypes.Sensor.ImageMsg [G:\unity_project\ros2test\Assets\RosMessages\Sensor\msg\ImageMsg.cs(11)]' to 'RosMessageTypes.Sensor.ImageMsg [G:\unity_project\ros2test\Library\ScriptAssemblies\Unity.Robotics.ROSTCPConnector.Messages.dll]'

I try to receive image data [ (ROS) usb_cam/image_raw] in unity and display it, and here's the problem:

Assets\cam_sub2.cs(28,55): error CS1503: Argument 1: cannot convert from 'RosMessageTypes.Sensor.ImageMsg [G:\unity_project\ros2test\Assets\RosMessages\Sensor\msg\ImageMsg.cs(11)]' to 'RosMessageTypes.Sensor.ImageMsg [G:\unity_project\ros2test\Library\ScriptAssemblies\Unity.Robotics.ROSTCPConnector.Messages.dll]'

And I generate the ROS message from /opt/ros/melodic/share/sensor_msgs.

Here is my code in unity

public class cam_sub2 : MonoBehaviour
{
    public RawImage showImage;
    void Start()
    {
        ROSConnection.GetOrCreateInstance().Subscribe<ImageMsg>("usb_cam/image_raw", image_callback);
    }

    void image_callback(ImageMsg img)
    {
        Texture2D tex = MessageExtensions.ToTexture2D(img,false,true,true);
        showImage.texture = tex;
    }
}

How can I resolve this?

Unity as the service node

Hi, for the current project I'm working on I need Unity to act as the service node, able to receive requests and send responses to ROS. I want to be able to call services using the service caller in RQT. This is something I was able to do with ROSSharp, however that functionality doesn't seem to exist in ROS-TCP-Connector. According to my understanding, the Unity service would need to advertise itself to ROS in order to be stored in the list of services, which can be called from the service caller. The SendServiceMessage function in the ROSConnection script seems similar to what I need however the other way around. Is there any way I can make a ReceiveServiceMessage function using the given tools? Thanks.

Messages backing up on Unity side

Hello,

I've been testing out this plugin, and am having issues with ROS messages getting backed up when they're received on the Unity side.

I'm publishing out messages from ROS @ 30hz where each message contains vectors of floats at least a few hundred values long.

What I observe is that the header sequence counter on the Unity side falls behind the ROS side, quickly reaching several seconds behind the ROS side. If I kill the ROS side, frames keep arriving in Unity for several seconds until it clears out what I assume is the TCP buffer.

This is all running on the same machine, so there's no bandwidth limitation. Is this expected behavior? Any idea what might be causing this and where we can start working on the code to fix it?

Thanks!

When I publish a jpeg image to unity this error appears in ROS terminal: struct.error: unpack requires a buffer of 5 bytes

ROS tcp endpoint does not allow me to publish jpeg image from Unity to ROS I'm using ros tcp endpoint v0.6.0
my problem main problem is that i want to publish unity camera image(Imagemsg) to ROS from an android device but the delay when i view in rviz is very high like 15-20 seconds using 1920 x 1080 resolution, so I tried to use send jpeg image (CompressedImagemsg) since jpeg is a highly compressed image, after I publish jpeg image i get this error from ROS .

struct.error: unpack requires a buffer of 5 bytes
raise genpy.DeserializationError(e) # most likely buffer underfill
genpy.message.DeserializationError: unpack requires a buffer of 5 bytes

how do i fix this?

This is my code for converting to jpeg

void SendJPEGImage(Texture2D _image)
{
byte[] data = _image.EncodeToJPG();

string picString = Convert.ToBase64String(data);
Debug.Log ("data length: " + data.Length);

// set format
string format = "jpeg";

//CompressedImageMsg(HeaderMsg header, string format, byte[] data)
    byte[] array = Encoding.ASCII.GetBytes(picString);
    HeaderMsg header = new HeaderMsg();
    var compressedImageMsg = new CompressedImageMsg(headerMessage, format, array);

    ros.Publish(topicName, compressedImageMsg);

}

Environment (please complete the following information, where applicable):

• Unity Version: [2020.3.34f1]
• Unity machine OS + version: [Windows 10]
• ROS machine OS + version: [Ubuntu 20.04, ROS Noetic]
• ROS–Unity communication: [ros-tcp-connector]
• Branch or version: [v0.6.0]

Additional context
please help my i have to finish this problem quick so I can proceed to another part in my project i've been stuck on this for a week please help me

Error during basic Unity Service tutorial

Describe the bug
Error calling UnityService, when following this section of the tutorial (using v0.5.0 release). I can see that the service has been successfully registered by ROS when Unity starts, but I get the following errors when I call rosservice call /obj_pose_srv Cube from the ROS PC. A similar error also occurs in the subsequent tutorial.

Console logs / stack traces
On Unity side:

Can't publish a message to an unregistered topic 'obj_pose_srv'
UnityEngine.Debug:LogError (object)
Unity.Robotics.ROSTCPConnector.ROSConnection:Send<Unity.Robotics.ROSTCPConnector.MessageGeneration.Message> (string,Unity.Robotics.ROSTCPConnector.MessageGeneration.Message) (at Library/PackageCache/com.unity.robotics.ros-tcp-connector@3f5d167040/Runtime/TcpConnector/ROSConnection.cs:731)
Unity.Robotics.ROSTCPConnector.ROSConnection/<>c__DisplayClass91_0:<ReceiveSysCommand>b__0 (string,byte[]) (at Library/PackageCache/com.unity.robotics.ros-tcp-connector@3f5d167040/Runtime/TcpConnector/ROSConnection.cs:463)
Unity.Robotics.ROSTCPConnector.ROSConnection:Update () (at Library/PackageCache/com.unity.robotics.ros-tcp-connector@3f5d167040/Runtime/TcpConnector/ROSConnection.cs:391)

On Ros side:

• Response to rosservice call command:

ERROR: service [/obj_pose_srv] responded with an error: error processing request: unpack requires a string argument of length 56

• in terminal running server_endpoint.py

rosout: Error processing request: unpack requires a string argument of length 56
['Traceback (most recent call last):\n', '  File "/opt/ros/melodic/lib/python2.7/dist-packages/rospy/impl/tcpros_service.py", line 632, in _handle_request\n    response = convert_return_to_response(self.handler(request), self.response_class)\n', '  File "/home/pioneer/catkin_ws/src/ROS-TCP-Endpoint/src/ros_tcp_endpoint/unity_service.py", line 53, in send\n    return self.tcp_server.send_unity_service(self.topic, self.service_class, data)\n', '  File "/home/pioneer/catkin_ws/src/ROS-TCP-Endpoint/src/ros_tcp_endpoint/server.py", line 99, in send_unity_service\n    return self.unity_tcp_sender.send_unity_service_request(topic, service_class, request)\n', '  File "/home/pioneer/catkin_ws/src/ROS-TCP-Endpoint/src/ros_tcp_endpoint/tcp_sender.py", line 106, in send_unity_service_request\n    response = service_class._response_class().deserialize(thread_pauser.result)\n', '  File "/home/pioneer/catkin_ws/devel/lib/python2.7/dist-packages/unity_robotics_demo_msgs/srv/_ObjectPoseService.py", line 226, in deserialize\n    raise genpy.DeserializationError(e)  # most likely buffer underfill\n', 'DeserializationError: unpack requires a string argument of length 56\n']

Expected behavior
As per the tutorial I expect to receive the position of the cube as a response to the service call.

Environment:

• Unity Version: Unity 2020.3.14f1
• Unity machine OS + version: Windows 10
• ROS machine OS + version: Ubuntu 18.04 (Running on Jetson Nano), ROS Melodic
• ROS–Unity communication: ROS-TCP-Connector (Non-docker-based) - communicating over ethernet.
• Branch or version: v0.5.0

Additional comment
Thanks so much to the team for your amazing work on ROS-Unity Integration and thanks to anyone that helps me solve this issue. Sorry if it's just me being stupid.

Threadsafe library

Is your feature request related to a problem? Please describe.
My robot's sensors use a library that runs on a different thread than the main thread, causing race conditions with the method void Send(), which throws the following exception:

TCPConnector Exception: System.Net.Sockets.SocketException (0x80004005): Only one usage of each socket address (protocol/network address/port) is normally permitted.

Describe the solution you'd like
It would be great if the ROSConnection class was threadsafe, or if it was possible to have multiple endpoints (one per thread).

Describe alternatives you've considered
I tried using lock() and a Semaphore but that did not help. I am not sure how to proceed.

Update 1: To ensure that only one thread could call the methods I am using (i.e. Send and RegisterPublisher), I created a wrapper that allowed all scripts to store their messages to a queue, and then only one endpoint was calling ProcessPublishers() but that did not work.

I also noticed the errors stats consistently at 75-80 seconds after the beginning of the simulation.

using Unity.Robotics.ROSTCPConnector;
using Unity.Robotics.ROSTCPConnector.MessageGeneration;
using System.Collections.Concurrent;

public class RosWrapper
{
    private static ConcurrentQueue<(string topicName, Message message)> msgBuffer = new ConcurrentQueue<(string, Message)>();

    private static ConcurrentQueue<(string topicName, string messageName)> pubRegBuffer = new ConcurrentQueue<(string, string)>();

    private static readonly object msgBufferLock = new object();

    public void Send(string rosTopicName, Message message)
    {
        msgBuffer.Enqueue((rosTopicName, message));
    }

    public void RegisterPublisher(string rosTopicName, string rosMessageName)
    {
        pubRegBuffer.Enqueue((rosTopicName, rosMessageName));
    }

    public void ProcessPublishers()
    {
        (string topicName, string messageName) regData;
        while(pubRegBuffer.TryDequeue(out regData))
            ROSConnection.instance.RegisterPublisher(regData.topicName, regData.messageName);

        (string topicName, Message message) pubData;
        while(msgBuffer.TryDequeue(out pubData))
            ROSConnection.instance.Send(pubData.topicName, pubData.message);
    }
}

Update 2: I decided to remove all the calls to the instance of ROSConnection and created a very simple "communication class" that just eases access to the message queues. Only one script is now allowed to use ROSConnection.instance. This implementation still does not work. The error continues to show up after 75-80 seconds.

using System.Collections.Concurrent;
using Message = Unity.Robotics.ROSTCPConnector.MessageGeneration.Message;

public class RosWrapper
{
    private static ConcurrentQueue<(string topicName, Message message)> msgBuffer = new ConcurrentQueue<(string, Message)>();

    private static ConcurrentQueue<(string topicName, string messageName)> pubRegBuffer = new ConcurrentQueue<(string, string)>();

    private static readonly object msgBufferLock = new object();

    public static void Send(string rosTopicName, Message message)
    {
        msgBuffer.Enqueue((rosTopicName, message));
    }

    public static void RegisterPublisher(string rosTopicName, string rosMessageName)
    {
        pubRegBuffer.Enqueue((rosTopicName, rosMessageName));
    }

    public static bool TryDequeueSend(out (string topicName, Message message) data)
    {
        return msgBuffer.TryDequeue(out data);
    }

    public static bool TryDequeuePubReg(out (string topicName, string messageName) data)
    {
        return pubRegBuffer.TryDequeue(out data);
    }
}

Update 3: I tried to create a thread dedicated to processing the information, but ROSConnector requires being executed on the main thread:

UnityException: Load can only be called from the main thread.

ros_tcp_endpoint: Cannot locate rosdep definition for [message_runtime]

Describe the bug
ros_tcp_endpoint: Cannot locate rosdep definition for [message_runtime]

To Reproduce
Clean install of ROS2 Foxy
git clone the Unity-Technologies/ROS-TCP-Connector
Move ROS-TCP-Connector folder to foxy/src folder
rosdep install --from-paths src --ignore-src --rosdistro foxy -y --skip-keys "console_bridge fastcdr fastrtps rti-connext-dds-5.3.1 urdfdom_headers"

Console logs / stack traces

ERROR: the following packages/stacks could not have their rosdep keys resolved
to system dependencies:
ros_tcp_endpoint: Cannot locate rosdep definition for [message_runtime]

Expected behavior
Expected to have the ROS-TCP-Connector installed

Environment (please complete the following information, where applicable):

• Unity Version: Unity 2020.3.11f1
• Unity machine OS + version: RHEL8.4
• ROS machine OS + version: Ubuntu 18.04, ROS Foxy
• ROS–Unity communication: kvm

Unable to install ROS-TCP-Connector

Describe the bug
I just follow the tutorial from：https://github.com/Unity-Technologies/ROS-TCP-Connector, want to install the ROS-TCP-Connector and use the Add package from git URL.... to install it, but it jumps out the error, and I also try to add the version like v0.4.0, v0.5.0, v0.6.0, v0.7.0, it still unable to install.
And I use the student version, so I don't know that is the student version can't download by URL or something.

Screenshots

Environment (please complete the following information, where applicable):

• Unity Version: Unity 2020.2.0f1
• Unity machine OS + version: Windows 10

Generates ROS1 messages even when set to ROS2

Describe the bug
I have created a local custom package with some msgs, srvs, and actions. I sourced it correctly, and it runs fine.

However, when i want to generate .cs files to use inside unity (going to Robotics -> Generate ROS Messages -> Build n msgs) the messages are generated alright, but they seem to be generated for ROS1. This became clear when i tried to use the class as type parameter to the ROS2.ISubscription. I then get the error message "The type 'RosMessageTypes.MyPkg.CustomMsg' must be convertible to 'ROS2.Message'".

I have set the protocol to ROS2 under Robotics -> ROS Settings -> Protocol.

I would expect the generated classes to extend the ROS2.Message class, but instead they extend a Message class from "Unity.Robotics.ROSTCPConnector.MessageGeneration".

Desktop:

OS: Ubuntu 20.04
ros2 distro: Galactic
ros2-for-unity version: Release 1.1.0
ROS-TCP-Connector: 0.7.0 Preview
ros2cs version: Unsure?
ros2 dds middleware: Default settings, i have not changed anything
ros2 environment setup: Local pc

Logic error when receiving data Unity Side

Hey there, I noticed an issue with the logic of receiving data on the unity side in the ROSConnection

dev branch
Line 216
and the issue also occurs on line 125 (duplicated code).

byte[] readBuffer = new byte[full_message_size];
int numberOfBytesRead = 0;

while (networkStream.DataAvailable && numberOfBytesRead < full_message_size)
{
    int bytesRead = networkStream.Read(readBuffer, 0, readBuffer.Length);
    offset += bytesRead;
    numberOfBytesRead += bytesRead;
}

The problem is that if the entire message isn't completely read in one Read, the next read will override the bytes read. Also it has the potential to read data from the next message.
Anyways, a fix could be something like this:

byte[] readBuffer = new byte[full_message_size];
int numberOfBytesRead = 0;
int bytesRemaining = full_message_size;
while (networkStream.DataAvailable && bytesRemaining > 0)
{
    int bytesRead = networkStream.Read(readBuffer, numberOfBytesRead, bytesRemaining);
    offset += bytesRead;
    numberOfBytesRead += bytesRead;
    bytesRemaining -= bytesRead;
}

Also I found that the
callback(msg);
Should probably have it's own try catch as if one subscriber throws an exception due to poor code subsequent subscribers won't receive the message. I mean ideally the callback(msg); should never throw an exception, but having it's own try catch would reduce the impact when a bug does occur and there are multiple subscribers for the same topic.

Oh and while I'm here, the same read code is used twice for both subscribers and services so it's really the first issue propagated twice so it could be improved if it uses the same code.

GetTopic doesn't work unless a topic exists at launch.

Describe the bug
Hi. I probably have a more complicated workflow vs. what you guys normally deal with, but I have an application that allows ROS as a possible input, but not a required input. As such, I don't necessarily have any ROS topics running on startup, or the topics I'm interested in, running when the application launches. I have found if I don't have the topics I'm interested in subscribing to running when the application launches, GetTopic (and the underlying m_topics list) do not have the topics I want in them.

Right now I am relying on a callback from GetTopicAndTypeList, which does return the up to date list, to set everything and subscribe. I had tried to do things like check the SentSubscriberRegistration and others for some state-related info, but I don't appear to be able to get that to work unless its there on startup.

To Reproduce
This is the gist of my setup:
///

/// The topic list buffer.
///

private Dictionary<string, string> topicList = new Dictionary<string, string>();

    public List<string> subscribedTopics;
    
    /// <summary>
    /// Dictionary of topic names and their types encoded as strings.
    /// </summary>
    public Dictionary<string, string> TopicListDictionary
    {
        get => topicList;
        set
        {
            if ( topicList == value )
            {
                return;
            }

            topicList = value;
            foreach (topic in topicList)
           {

               if (!subscribedTopics.Contains(topic.Key))
               {
                    subscribedTopics.Add(topic.Key);
                    if ( ROSConnectionInstance.GetTopic( topic.Key ).SentSubscriberRegistration )
                    {
                        continue;
                    }

                    ROSConnectionInstance.Subscribe<RosMessageTypes.Sensor.PointCloud2Msg>(topic.Key, TransferPcl2Points);
               }
           }
        }
    }
    
    private void Update()
    {
            // I use some timer here to limit this but for the example this works
            PerformRosUpdate();
    }

    /// <summary>
    /// Performs the update of ROS connection status and topics.
    /// </summary>
    /// <returns></returns>
    private void PerformRosUpdate()
    {
        if ( !ROSConnection.GetOrCreateInstance().HasConnectionError )
        {
            ROSConnection.GetOrCreateInstance().GetTopicAndTypeList( CopyTypesToList );
        }
    }

    private void TransferPcl2Points(RosMessageTypes.Sensor.PointCloud2Msg pcl2Points)
    {
        //unrelated;
    }
    
    /// <summary>
    /// A callback to copy the topic list with their types from ROS to here.
    /// </summary>
    private void CopyTypesToList( Dictionary<string, string> topicsAndTypes )
    {
        TopicListDictionary = topicsAndTypes;
    }

Console logs / stack traces
Please wrap in triple backticks (```) to make it easier to read.

Expected behavior
If the topic exists before I've run my application, the above works great. If it doesn't, the application crashes because if ( ROSConnectionInstance.GetTopic( topic.Key ) returns NULL.

Screenshots
If applicable, add screenshots or videos to help explain your problem.

Environment (please complete the following information, where applicable):

• Unity Version: [e.g. Unity 2020.3.21f1]
• Unity machine OS + version: [e.g. Ubuntu 18.04]
• ROS machine OS + version: [e.g. Ubuntu 18.04, ROS Melodic]
• ROS–Unity communication: [Same machine]
• Branch or version: ROS-TCP-Endpoint-0.7.0

OnApplicationQuit should close the tcp connection

I might got this wrong but after I close the app it does not properly stop the connection and some aspects like moving behavior continues even after the game was stopped.

Maybe I am doing something wrong but I also cannot see that the connection is closed in the main ROSConnection.cs script.

Love the package. Thank you all for developing this.

Installation of ROS-TCP-Connector fails via Unity PackageManager due to missing package manifest file

Environment:

• Unity 2021.1b
• MacOS Catalina (10.15.7)

As per the installation instructions, I attempted installing the following git URL (via the Unity Package Manager):
https://github.com/Unity-Technologies/ROS-TCP-Connector.git

However, the following errors show up:

[Package Manager Window] Cannot perform upm operation: Unable to add package [https://github.com/Unity-Technologies/ROS-TCP-Connector.git]:
  [https://github.com/Unity-Technologies/ROS-TCP-Connector.git] does not point to a valid package repository. No package manifest was found. Verify the repository URL and make sure the package is located in the root folder of the repository. [NotFound].
UnityEditor.EditorApplication:Internal_CallUpdateFunctions () (at /Users/bokken/buildslave/unity/build/Editor/Mono/EditorApplication.cs:327)

[Package Manager Window] Error adding package: https://github.com/Unity-Technologies/ROS-TCP-Connector.git.
UnityEditor.EditorApplication:Internal_CallUpdateFunctions () (at /Users/bokken/buildslave/unity/build/Editor/Mono/EditorApplication.cs:327)

PointCloud2 Visibility in VR

Describe the bug
I have suceeded in using the new Visualisations package to plot a pointcloud within my scene, however when viewing the scene in VR, the point cloud is invisible in both the 'Game' view and within the HMD. However, when looking at the 'Scene' view, I can see the point cloud.

To Reproduce
Docker container running pointcloud from .bag file and ROS-TCP-Connector. Unity 2020.3.26f1 with Visualisations package. When viewing in VR mode, the pointcloud topic can be selected in the topics list, and viewable in 'Scene' view but not in HMD or 'Game' view.

Console logs / stack traces
No error messages.

Expected behavior
Point cloud 'dots' visible in VR.

Screenshots
If applicable, add screenshots or videos to help explain your problem.

Environment (please complete the following information, where applicable):

• Unity Version: 2020.3.26f1
• Unity machine OS + version: Windows 10
• ROS machine OS + version: Dockerised ROS Noetic
• ROS–Unity communication: ROS-TCP-Endpoint & Connector

Is it possible to use the visualization package to display the trajectory?

Can the visualization package be used to display the trajectory the robot completes?
I know this may sound stupid but I would like a line to appear tracking the end effector in the pick and place tutorial as in RViz and I can not understand if this is possible through this package. If it is can you please explain to me which passages I should follow, I tried the DefaultVisualizationSuite but the Drawing manager keeps throwing errors that I can not understand is I am a newbie still.

ActionServer and ActionClient ROS Support

Feature Request: At present Action server and client modules are not supported in the ros unity connection. As a result of this, many manipulator services are not enabled.

Display data from stereo camera

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
The Intel D435 sends different image frames for left and right camera. How do we display these in the Unity scene for the Oculus Quest? Would this be termed 2d or 3d?

Describe the solution you'd like
A clear and concise description of what you want to happen.
It won't be hard to display both images side by side, but is this what we want to do in order to send the video to each eye in the headset? The human viewer makes it into 3d!

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.
I want it to seem like the headset wearer is "inside" the robot, seeing just what its camera is seeing. I realize the scene won't be as tall as the Oculus wants (high resolution) but the effect may still be good.

Additional context
Add any other context or screenshots about the feature request here.
I will implement this with head tracking that sends commands to the robot's pan and tilt camera mount.

CameraInfo Generator

Is your feature request related to a problem? Please describe.
In ROS, when working with Image messages (particularly stereo cameras), the CameraInfo topic is important for nodes such as stereo_image_proc and visual odometery. For the past two projects I've written some code to automatically generate CameraInfo for unity cameras. I was considering writing a more general version and adding the code to the ROS-TCP-Connector.

Describe the solution you'd like
Add a CameraInfo generator based on Unity Camera inputs.

Describe alternatives you've considered
Re-writing the code for every project.

Additional context
I understand that this project is mainly focused on the Connection between ROS and Unity, so maybe this is more of a utility that should go into a utility repository that's Unity ROS related, but not specifically the connector, does Unity have one for such things?
Note: I checked Unity-Robotics-Hub, but didn't see anything suitable.

ROS TCP Connector doesn't work on my Oculus Quest 2

Hi Everyone,

I'm trying to finish my end-of-degree project, which consists of doing some VR tasks with NAO Robot on Oculus Quest 2 and Unity. To do that, I have some scripts in ROS, so my aim is to connect Unity and ROS. So far I have managed to connect Unity and ROS successfully using ROS-TCP-Connector. However, now that I'm trying to run the app on the Oculus Quest 2, after I build and run it, nothing happens. I can see the scene with the robot and I get no compilation errors, however, it seems that the connection between Oculus and ROS never happens.

I tried another thing in order to see if ROS was the problem. I created a script in python which opens a server and waits for Unity to act as client. With this same ROS-TCP-Connector, when I specify the server ip (the one I created in python) and I run the app in Unity, I can see in the terminal that something connected to the server. However, again, when I build it and run it from the Oculus, the server stays waiting because nothing connects to it.

I don't know if any of you can help or advice me on how to continue. I have searched on the internet and I have tried a lot of things but any of them worked at all.

Thank you so much in advance!

ROS Time

Is your feature request related to a problem? Please describe.
I just realized that ROS-TCP-Connector does not fill in the stamp field of the Header, so all stamped messages are submitted without time information.

Describe the solution you'd like
Either synchronize Unity with ROS (maybe using NTP) or generating the simulation time and publishing it to /clock.

Describe alternatives you've considered
Right now I am implementing both cases. I installed an NTP server on Ubuntu 18.04 (ROS Melodic) and adapted this simple NTP client to synchronize Unity and ROS.

For scenarios where the simulation time can be used, I publish rosgraph_msgs/Clock messages to /clock.

Additional context
ROS time is represented by two components: secs for seconds and nsecs for nanoseconds. I have been using Unity's Time.timeAsDouble to generate the ROS time, but I wonder if that is the best way to do it.

Cannot build Unity project for HoloLens (UWP)

Hi, I'm working on a simple Robotics project where I need to control a virtual robot (in Unity) through HoloLens.
I've created a simple test scene with the imported URDF model of the robot itself (Universal Robot UR5) and installed the library ROS-TCP-connector as suggested from the installation guide.
Just for some tests, I've tried to import a simple JointState message.

If I build the project for the PC architecture, no problems at all, but when I select the Universal Windows Platform from the build preferences of Unity, the editor launches the following error messages:

• Assets\RosMessages\Sensor\msg\MJointState.cs(8,13): error CS0234: The type or namespace name 'Robotics' does not exist in the namespace 'Unity' (are you missing an assembly reference?)
• Assets\RosMessages\Sensor\msg\MJointState.cs(58,38): error CS0115: 'MJointState.SerializationStatements()': no suitable method found to override
• Assets\RosMessages\Sensor\msg\MJointState.cs(82,29): error CS0115: 'MJointState.Deserialize(byte[], int)': no suitable method found to override
• Assets\RosMessages\Sensor\msg\MJointState.cs(127,32): error CS0115: 'MJointState.ToString()': no suitable method found to override

I'm quite new to Unity environment and I believe that this problem can be caused by some compiling options or preprocessing directives that I am not aware of.

The environment in this study case is the following:

• Unity 2020.3.9f1
• Windows 10 (PC with Unity and ROS melodic installed)
• Microsoft HoloLens (target architecture)
• .NET 4.0 build option

Standard on whether .meta files should be added to the git repo.

Hi, not really a feature request, but after adding this repo as a submodule to a git project, currently on the dev branch, I have found that Unity (Using 2021.1.14f1) has automatically generated a number of .meta files that aren't covered by the .gitignore.

I just would like to know what the standard here is, should all .meta files be .gitignored, if not which ones should be added?

For further reference, here are all the .meta files Unity has added but are not tracked.

[Expandable] .meta files added but not tracked:

• CODE_OF_CONDUCT.md.meta
• CONTRIBUTING.md.meta
• InfoAcknowledgements.md.meta
• LICENSE.meta
• MessageGeneration.md.meta
• README.md.meta
• ROSGeometry.md.meta
• TestRosTcpConnector.meta
• TestRosTcpConnector/Assets.meta
• TestRosTcpConnector/Packages.meta
• TestRosTcpConnector/Packages/manifest.json.meta
• TestRosTcpConnector/Packages/packages-lock.json.meta
• TestRosTcpConnector/ProjectSettings.meta
• TestRosTcpConnector/ProjectSettings/AudioManager.asset.meta
• TestRosTcpConnector/ProjectSettings/ClusterInputManager.asset.meta
• TestRosTcpConnector/ProjectSettings/DynamicsManager.asset.meta
• TestRosTcpConnector/ProjectSettings/EditorBuildSettings.asset.meta
• TestRosTcpConnector/ProjectSettings/EditorSettings.asset.meta
• TestRosTcpConnector/ProjectSettings/GraphicsSettings.asset.meta
• TestRosTcpConnector/ProjectSettings/InputManager.asset.meta
• TestRosTcpConnector/ProjectSettings/NavMeshAreas.asset.meta
• TestRosTcpConnector/ProjectSettings/PackageManagerSettings.asset.meta
• TestRosTcpConnector/ProjectSettings/Physics2DSettings.asset.meta
• TestRosTcpConnector/ProjectSettings/PresetManager.asset.meta
• TestRosTcpConnector/ProjectSettings/ProjectSettings.asset.meta
• TestRosTcpConnector/ProjectSettings/ProjectVersion.txt.meta
• TestRosTcpConnector/ProjectSettings/QualitySettings.asset.meta
• TestRosTcpConnector/ProjectSettings/TagManager.asset.meta
• TestRosTcpConnector/ProjectSettings/TimeManager.asset.meta
• TestRosTcpConnector/ProjectSettings/URPProjectSettings.asset.meta
• TestRosTcpConnector/ProjectSettings/UnityConnectSettings.asset.meta
• TestRosTcpConnector/ProjectSettings/VFXManager.asset.meta
• TestRosTcpConnector/ProjectSettings/VersionControlSettings.asset.meta
• TestRosTcpConnector/ProjectSettings/XRSettings.asset.meta
• Third Party Notices.md.meta
• com.unity.robotics.ros-tcp-connector.meta

Closing the TCP connection when not playing

Hi, I've been using this for a couple of weeks now and it's been great. Thank you for creating it!

I've recently run into this problematic behavior with the connection and made a fix for it.

Currently, when the Application quits (when the user stops playing the scene in the editor), the TcpListener in RosConnection.cs continues to process incoming messages and call registered subscribers.

This is a problem if one has stopped playing in the editor and then does a "Build and Run". In this case the running application will throw an exception and not connect to the ROS server because the TcpListener which was started in the editor has not released the address.

Here's a fix that works for me. It's checking Application.isPlaying in the outer loop of StartMessageServer() in RosConnection.cs to prevent the code from creating a TcpListener when the application is not playing.
And, it's checking Application.isPlaying in the inner loop so that it can call Stop() the TcpListener, set it to null and break out of the inner loop.

Also, I think this is related to #20

Here's the modified version of StartMessageServer():

` protected async void StartMessageServer(string ip, int port)
{
if (alreadyStartedServer)
return;

    alreadyStartedServer = true;
    while (true)
    {
        if (!Application.isPlaying)
        {
            await Task.Yield();
        }
        else
        {
            TcpListener tcpListener;
            try
            {
                tcpListener = new TcpListener(IPAddress.Parse(ip), port);
                tcpListener.Start();
            }
            catch (Exception e)
            {
                Debug.LogError("Exception raised!! " + e);
                return;
            }

            Debug.Log("ROS-Unity server listening on " + ip + ":" + port);

            try
            {
                while (true)   //we wait for a connection
                {
                    if (!Application.isPlaying)
                    {
                        if (tcpListener != null)
                        {
                            tcpListener.Stop();
                            tcpListener = null;
                            await Task.Yield();
                        }
                        break;
                    }

                    var tcpClient = await tcpListener.AcceptTcpClientAsync();

                    var task = StartHandleConnectionAsync(tcpClient);
                    // if already faulted, re-throw any error on the calling context
                    if (task.IsFaulted)
                        await task;

                    while (tcpListener.Pending())
                    {
                        tcpClient = tcpListener.AcceptTcpClient();
                        task = StartHandleConnectionAsync(tcpClient);
                        if (task.IsFaulted)
                            await task;
                    }
                }
            }
            catch (Exception e)
            {
                Debug.LogError("Exception raised!! " + e);
            }
        }
    }
}

`

ArgumentOutOfRange while deserializing moveit_msgs/DisplayTrajectory message

Describe the bug
I sent a moveit_msgs/DisplayTrajectory from ROS2 to Unity but got an Error ArgumentOutOfRangeExeption: Non-negative number required.

Message

model_id: ur
trajectory:
- joint_trajectory:
    header:
      stamp:
        sec: 0
        nanosec: 0
      frame_id: base_link
    joint_names:
    - shoulder_pan_joint
    - shoulder_lift_joint
    - elbow_joint
    - wrist_1_joint
    - wrist_2_joint
    - wrist_3_joint
    points:
    - positions:
      - 0.0
      - 0.0
      - 0.0
      - 0.0
      - 0.0
      - 0.0
      velocities:
      - 0.0
      - 0.0
      - 0.0
      - 0.0
      - 0.0
      - 0.0
      accelerations:
      - 0.0
      - 0.0
      - 0.0
      - 0.0
      - 0.0
      - 0.0
      effort: []
      time_from_start:
        sec: 0
        nanosec: 0
  multi_dof_joint_trajectory:
    header:
      stamp:
        sec: 0
        nanosec: 0
      frame_id: ''
    joint_names: []
    points: []
trajectory_start:
  joint_state:
    header:
      stamp:
        sec: 0
        nanosec: 0
      frame_id: base_link
    name:
    - shoulder_pan_joint
    - shoulder_lift_joint
    - elbow_joint
    - wrist_1_joint
    - wrist_2_joint
    - wrist_3_joint
    - finger_joint
    - left_inner_finger_joint
    - left_inner_knuckle_joint
    - right_inner_knuckle_joint
    - right_outer_knuckle_joint
    - right_inner_finger_joint
    position:
    - 0.0
    - 0.0
    - 0.0
    - 0.0
    - 0.0
    - 0.0
    - 0.0
    - 0.0
    - 0.0
    - 0.0
    - 0.0
    - 0.0
    velocity:
    - 0.0
    - 0.0
    - 0.0
    - 0.0
    - 0.0
    - 0.0
    - 0.0
    - 0.0
    - 0.0
    - 0.0
    - 0.0
    - 0.0
    effort: []
  multi_dof_joint_state:
    header:
      stamp:
        sec: 0
        nanosec: 0
      frame_id: base_link
    joint_names: []
    transforms: []
    twist: []
    wrench: []
  attached_collision_objects: []
  is_diff: false
---

Screenshots
Error

The "length" value is equal 0 while deserializing /trajectory/multi_dof_joint_trajectory/header/frame_id .

Environment (please complete the following information, where applicable):

• Unity Version: Unity 2020.3.13f1
• Unity machine OS + version: Windows 11
• ROS machine OS + version: Ubuntu 20.04, ROS Foxy
• ROS–Unity communication: Docker
• Branch or version: v0.6.0

Additional context

Many thanks in advance !

Ros2 Action support

Is your feature request related to a problem? Please describe.
I am currently using both messages and services without any issues. I am, however, unsure of whether TCP-connector supports ROS2 actions. It does say anywhere, that it doesn't. However, i cannot find the methods for using them in the ROSConnection class. I am also able to generate the action cs files from my .action files, so this suggests to me, that actions are supported.

Describe the solution you'd like
Action support in the ROSConnection. A way to register for actions with callback on finish and a way to publish on the feedback channel easily.

Describe alternatives you've considered
Simply using messages and create a channel for feedback and cancelling and action myself. This would mimic the functionality of actions.

Additional context
Add any other context or screenshots about the feature request here.

Namespace issues with using Unity.Robotics.Visualizations

Describe the bug
When I try to create my own custom visualizations, I am getting namespace errors. I'm not sure how to properly include the dll to reference the Robotics Visualization package.

To Reproduce

1. Import packages URDF Importer, ROS-TCP-Connector and Robotics Visualizations
2. Create a new script using examples from Visualizations.md.
3. Change namespace from using Unity.Robotics.MessageVisualizers to using Unity.Robotics.Visualizations

Console logs / stack traces

The type or namespace name 'Visualizations' does not exist in the namespace 'Unity.Robotics' (are you missing an assembly reference?)

Expected behavior
I should be able to reference the functions within the namespace Unity.Robotics.Visualizations

Environment (please complete the following information, where applicable):

• Unity Version: [e.g. Unity 20203.28f1]
• Windows 11
• v0.7.0

ROSService crashed after repeated calls

Hi, I'm implementing a ServiceServer in Unity. I launch the ros_tcp_endpoint.launch and start my unity simulation. The first time a start a client ros node that call the services it works usually fine. However, if I restart the client node again. The ros_tcp_endpoint outputs this error.

[ERROR] [1643902287.677552]: Not registered to publish topic 'set_lights'! Valid publish topics are: dict_keys(['unity_camera/result', 'unity_camera/feedback', 'unity_camera/status', 'model_states', '/robot_collision', 'tool_joint_states', '/trajectory_action/result', '/trajectory_action/feedback', 'joint_states', '/trajectory_action/status']) 

set_lights is the name of the service. This behavior happens randomly, sometimes the first time a call the node, sometimes the second, the third...

I just tried the same code in the release version v.0.6.0 and it works fine, so It is only a problem of the dev branch

Environment

• Unity Version:Unity 2020.3.21f1
• Unity machine OS + version: Ubuntu 20.04
• ROS machine OS + version: ROS Noetic
• Branch or version: dev

Visualizing data from the robotic arm's camera

Bug/Error:
I added the Visualizations Package system you specified to unity, but I can't find a button to visualize the frames when the robotic arm is moving in the HUD window. In short, I can't watch streaming camera footage. Can you help?

Expectation:
I want to visualize the images from the camera while the robotic arm is operating.

Environment :

• Unity 2020.3.0f1
• Ubuntu 20.04, ROS Noetic
• Docker

Can this method publish image to ros?

ROS subscriber uses image_transport and cv_bridge to receive images, BUT I cannot find similar tutorials or modules to publish unity-cam pictures, so can you help me?

Compile Errors when Switching to WebGL

Some of the method and class signatures change when switching unity platform to WebGL.
For example:
public TimeMsg(int sec, uint nanosec);
becomes
public TimeMsg(uint sec, uint nanosec);

ROS msgs not received in a topic if publisher starts after subscriber

Main bug description
If I subscribe to a topic before a publisher was registered at that topic, no ROS msgs are received by the subscriber callback once the publisher is registered.

To Reproduce
Steps to reproduce the behavior:

1. Run 'rostopic echo SOME_TOPIC'
2. Run an application (application A) with a publisher registered to a certain topic. E.g.:

using System.Collections;
using System.Collections.Generic;
using RosMessageTypes.Std;
using Unity.Robotics.ROSTCPConnector;
using UnityEngine;

public class SimpleSender : MonoBehaviour
{
    ROSConnection ros;

    public ROSConnection Ros
    {
        get => ros;
        set => ros = value;
    }

    public string value;
    
    public string Value
    {
        get => value;
        set => this.value = value;
    }

    public string topicName;
    public string TopicName
    {
        get => topicName;
        set => topicName = value;
    }
    
    public void Start()
    {
        // start the ROS connection
        Ros = ROSConnection.GetOrCreateInstance();
        Ros.RegisterPublisher<StringMsg>(TopicName);
    }

    public void Send(string value)
    {
        this.value = value;
        var message = ConvertToRosMessage(value);
        Ros.Publish(TopicName, message);
        Debug.Log("Value sent");
    }

    public StringMsg ConvertToRosMessage(string value)
    {
        var message = new StringMsg();
        message.data = value;
        return message;
    }

}

3. Run another application (application B) with a subscriber registered to the same topic. E.g.:

using System.Collections;
using System.Collections.Generic;
using RosMessageTypes.Std;
using Unity.Robotics.ROSTCPConnector;
using UnityEngine;
using UnityEngine.Events;

[System.Serializable]
public class SimpleReceiverEvent : UnityEvent<string>
{
}

public class SimpleReceiver : MonoBehaviour
{
    ROSConnection ros;
    public StringReceiverEvent OnChange;

    public ROSConnection Ros
    {
        get => ros;
        set => ros = value;
    }
    
    public string topicName;
    public string TopicName
    {
        get => topicName;
        set => topicName = value;
    }
    
    private void Start()
    {
        Ros = ROSConnection.GetOrCreateInstance();
        Ros.Subscribe<StringMsg>(TopicName, MessageChange);
    }
    
    void MessageChange(StringMsg message)
    {
        Debug.Log("ROS msg received");
        var value = ConvertFromRosMessage(message);
        Receive(value);
    }
    public void Receive(string value)
    {
        OnChange.Invoke(value);
    }
    public string ConvertFromRosMessage(StringMsg message)
    {
        var value = message.data;
        return value;
    }
}

3. When the method 'Send' from 'SimpleSender' in 'application A' is invoked, the message can be seen both using 'rostopic echo' and is properly received in application B (e.g., 'Debug.Log' is properly invoked.
4. Now close the applications, leave 'rostopic echo' running and run the applications in the opposite order: first run application B then run application A.
5. When the method 'Send' from 'SimpleSender' in 'application A' is invoked: application B does not receive the method; but, 'rostopic echo' does receive the message, which shows that it was properly published.

After several tests we concluded that, in TCP Connector, if a publisher isn't registered before a subscriber then the subscriber does not receive the messages from the publisher.

Expected behavior
We expect publishers and subscribers registered to the same topic to communicate between themselves independently of the order in which they were created.

Instead, if I subscribe to a topic before a publisher was registered at that topic, no ROS msgs are received by the subscriber callback once the publisher is registered.

For the sake of simplicity let's call A the machine that is publishing and B the machine that is subscribing to the topic.

If I run 'rostopic echo NAME_OF_TOPIC', I can see that the messages are indeed being published by A. However, the 'subscriber' in B does not seem to be able to receive them if B is run after A.

I came up with a couple of "hacks" to handle the issue. If I set a timeout in B when listening for msgs on a subscriber and either: disconnect/reconnect to RosConnection; or, unsubscribe/resubscribe to the topic; then, the topic is properly subscribed (in B) and receives messages (from A).

However, I've verified that these solutions have the side-effect that publishers in B lose "connection" to their related subscribers in other machines (e.g., in A). I also verified that to unsubscribe/resubscribe to the topic takes a long time (in the order of seconds), which means that messages get lost during that time, and B may not receive a message from A within time, failing the timeout check again.

For completeness, I'm using the following to register a publisher:

Environment:

• Unity Version: 2021
• Unity machine OS + version: Ubuntu 18.04
• ROS machine OS + version: same as Unity machine
• ROS–Unity communication: Ros and ROS-TCP-Endpoint running natively on the machine
• Branch or version: TCP Connector and Endpoint version 0.6.0

Unable to install ros-tcp-visualizer

I just follow the tutorial from https://github.com/Unity-Technologies/ROS-TCP-Connector/blob/v0.6.0/com.unity.robotics.visualizations/Documentation~/README.md, but my unity continues reporting errors, just shown in the image.

How do we solve it? I searched previous issues and ros unity forum but still got nothing.