introlab / opentera-webrtc-teleop-frontend

OpenTera VUE3.js Frontend to be used with opentera-webrtc signaling server (local or on the Internet)

opentera-webrtc-teleop-frontend

An OpenTera web user interface to teleoperate a robot. Made with Vue.js 3.

Authors

• Gabriel Lauzier (@G-Lauz)
• Marc-Antoine Maheux (@mamaheux)
• Jérémie Bourque (@JeremieBourque1)
• Dominic Létourneau (@doumdi)
• Philippe Warren (@philippewarren)
• Ian-Mathieu Joly (@joli-1801)

Features

• Dual camera view
• Virtual joystick to move the robot
• Use the arrow keys to move the robot
• A slider to limit the maximum speed
• Battery and wifi signal indicators
• A draggable and expandable map widget which can be zoomed and panned with the mouse wheel or touch gestures
• Ability to place waypoints on the map that the robot can navigate to autonomously
• Autonomous docking
• Option to toggle between the mapping and localization modes.

Demo

The easiest way to try out the interface is to use the Gazebo demo in opentera-webrtc-ros. You can find the instructions on how to run the demo here

Related projects

• opentera
• opentera-webrtc
• opentera-webrtc-ros
• opentera-teleop-service

License

• Apache License, Version 2.0

Sponsor

IntRoLab - Intelligent / Interactive / Integrated / Interdisciplinary Robot Lab

opentera-webrtc-teleop-frontend's Issues

Opentera-webrtc controller

Implement button and there action in order to interact with the opentera-webrtc-teleop web client librairy.

• Call all
• Hang up
• Mute
• Close / open camera

Video conference client prioritization on small display

Create an algorithm to prioritize clients in order to display those with higher priority when the display his to small for multiple clients.

The device participating in a conference should have an higher screen priority.

Potential solutions:

• Prioritize according to the amplitude of sounds track of the clients
• Prioritize clients with their camera / mic open

Create different type of ClientStore

The ClientStore create all the instances of SignalingClientStore module needed in the worst case. However, we have two components (DefaultTemplate and ConferenceView) that don't necessary needs all those instances. As example, ConferenceView only needs an instance of StreamClientStore to work properly. This behaviour make the code hard to read, to debug and make it difficult to add new features.

A way to get around this issue could be by using the Builder design pattern where ClientStore will be refactor in a class like store like SignalingClientStore. That way, we can dynamically create the ClientStore module with its specifications in an actions of the root store (See Dynamic module registration).

This issue will also solve the ambiguity in the action toggleCall of ClientStore. At the moment we don't know which type of client try to make a call. If a device client which only need one instance of StreamClientStore call toggleCall it will try to call with every instance possible of SignalingClientStore even if they haven't been initialize. It make it hard to handle possible error.

Disable toolbar buttons until the stream has started

You can click on buttons such as mute and toggle camera in the toolbar before the stream has started causing errors. They should be disabled until the stream has started.

Make WaypointOverlay more efficient

WaypointOverlay draws the canvas continously at a fixed rate (30 Hz be default) even when there are no changes. This is very inefficient and uses the cpu more than it should. The canvas should only be drawn when there is a change.

Refactor store

Instead of creating a complex inheritance structure with Vuex modules in signalingClientStore.ts and using its instances as the modules of clientStore.ts we can create an inheritance structure without Vuex in it and use its instances as the state of clientStore.ts.

Doing this we will have less store namespace in the code and it will be easier to debug and to add features.

Add the possibility to slect which input device to use

Add settings that allow the users to select which camera nor microphone they want to use.

Keyboard teleop publishing frequency slow for first message sent

The first message sent when using the keyboard arrows to teleoperate the robot has a longer pause than the following messages. This causes an issue with the twist multiplexer because the delay is long enough to fall back to the zero twist publisher causing a jitter in the robots movement at the start. The problem also appears if we press a second arrow key while still pressing the first one.

We want the publishing rate to always be the same.

Demo interface with three different routes

• robot route (screen on the robot displaying video streams)
• operator route (allowing remote control of the robot)
• participant route (external read-only client)

A room will be created by the operator and everybody will be invited to join the session.

Inspiration from :

• (private) : https://github.com/introlab/agewell/tree/webrtc_victoria
• (public) : https://github.com/introlab/securbot

TODO:

• Create Vue.js reusable components
• The application will be served in the "static" files of the signaling server
• Use opentera-webrtc-teleop Javascript library to connect to signaling server
• Use ROS for the robot
• Use a web browser for the operator

Build / copy / adapt session (dynamic with query args) frontend to work with opentera-teleop-service signaling-server.

The generated project must be copied to signaling-static folder of the signaling-server, integrated to the opentera-teleop-service.

• Teleoperation with a virtual joystick or keyboard.
• Manage "rooms" to optimize bandwidth.
• Allow invitation of "guests".
• Allow display of the robot's status (batteries, state, etc.)

Video Conference Widget

Create a responsive video conference widget
TO DO:

• Style of the component
• Use Vue.js 3 Composition API
• Responsivity of the component
  • Main speaker get the fullscreen on small screen or in card mode
  • Give screen priority to the robot stream
  • Set a maximum clients per Media Query
• Create two mode of usage (Fullscreen / constraint)

Change layout according to the type of robot to teleoperate

IntRoLab - We need to support both modified beam robots and TTOP robot.

• Available actions should change
• Camera layout should change
• Virtual Joystick should be enabled / disabled
• Map should be enabled / disabled

Define a maximum number of users per Media Query

To many user on a screen can create confusion and annoy the users.

The goal his to find the maximum number of user we should display according to size of the screen.

If the number of user his higher than the limit then we should use an algorithm as describe in this issue: #12

Add opentera-teleop-service frontend for robot fleet management.

This will require to connect to OpenTera's API and teleop service API to manage online robots, and start sessions.

• APIs will be defined by @doumdi.
• Client authentication and authorizations must be used.
• Use VUE.js and components.
• The single page app will be served in the static folder of the service.
• Installation guides and documentation

Evaluate the replacement of custom JSON and websockets with rosbridge and roslibjs

See this issue: introlab/opentera-webrtc-ros#32

Joystick sometimes disapears after a while

The joystick is only drawn during a mousedown and mousemove event. It is not constantly drawn because that would use up a lot of ressources, which is not very efficient. However, after some time of inactivity, the joystick disappears and only reappears when you click on it.

Add maximum speed slider

Add a slider to choose the maximum speed of the robot in the interface.

Update opentera-webrtc submodule to latest version.

After introlab/opentera-webrtc#106 is merged.

Delay on client deconnection

There's a short delay for the others clients in a call when a client hangs up. The image of the client that has disconnect his frozen for a few seconds and then disappear as it should behave.

We didn't have a delay on previous version: Update README (#10). Looks like the refactor made in PR #11 break it.

The same behaviour can be witnessed in the examples of opentera-webrtc

Router view application

Create the base layout of the web application using Vue.js 3 Router view.

Setup css color inheritage

Setup a way to easily change the color of the web app,

Teleoperation widget

Create a Teleoperation widget with the following features:

• Display the front camera of the teleoperation device (device stream in VideoConf room)
• Display the bottom camera of the teleoperation device (device stream in CameraX room)
• A virtual joystick nor keyboard input for navigation (Teleop room)
• Display the user with the higher priority in the VideoConf room
• A chat (Messaging room)
• Display miniatures of the participants with lower priority (VideoConf room)

Take a look at the architecture for room management: