virtigex / r2k9_telepresence Goto Github PK

The aim of this project is to develop a telepresence and teleoperation application in the robotic field using open source software
This application has been done by Oier Mees during a summer internship at Tekniker Research Center and is now maintained by Ane Fernandez <[email protected]>

Used tools:
  Hardware
    Segway rpm 200 robot (with following sensors: laser range finder, thermopile, sonar, bumper, encoder, stargazer IR to read tags in indoor navigation)
    Kinect camera on top of the robot
    Webcam for the client
    PS3 Controller

  Software
   ROS http://www.ros.org diamondback (also works with electric)
   Ubuntu  10.04 Lucid
   Opencv2 stack for the image handling
   PS3Joy stack for the PS3 controller 
   Openni stack for the kinect
   audio_capture stack to capture and stream the webcams microphones sound


How to compile and run the application
--------------------------------------
First of all, configure ROS for multiple machines by following the next tutorials:
http://www.ros.org/wiki/ROS/NetworkSetup
http://www.ros.org/wiki/ROS/Tutorials/MultipleMachines
It is important to change the $ROS_MASTER_URI variable and that both machines can resolve the name of the other (check /etc/hosts file).

on the robot:
  1) enable navigation mode, depends on the robot
  2) start kinect topic with 
     $ rosrun openni_camera openni_node
  3) optional: if you are streaming the video over wifi, you may want to resize the image to decrease the latency using the code kindly provided by José María Martínez Otzeta
     $ rosrun tekniker_telepresence image_resizer  50 5 /image:=/camera/rgb/image_color
     in this case, the launcher file of the client has to be changed  to remap the rgb topic by uncommenting the following line
     <remap from="/camera/rgb/image_color" to="/image_resized"/>
  4) start the depth service, which will return the mean depth (x,y) coordinates in the kinect
     $ rosrun tekniker_kinect depth_server
  5) optional: if you don't have an actual robot you can always use a simulator (the following packages are not provided)
      $ roslaunch tekniker_simulation tmm_stage.launch
      $ roslaunch 2dnav_segway_rmp move_base_tmm_simulator.launch
  6) After completing the steps of the client machine, you have to launch audio_play to play the sound that is streamed over the /audio topic
      $ rosmake audio_play
      $ roslaunch audio_play play.launch


on the client machine:
  1) enable ps3 controller, see http://www.ros.org/wiki/ps3joy/Tutorials/PairingJoystickAndBluetoothDongle
     $ sudo bash
     $ rosrun ps3joy ps3joy.py
     press the pairing button. If it doesn't activate, try the methods at http://www.ros.org/wiki/ps3joy/KarmicInstructions
     After pairing the controller, you have to press the button with the rectangle to activate joystick teleoperation and
     press it again to disable it. Then you can enable gyro teleoperation by pressing the button with the triangle.
     Remember always, that the robot can't be navigating to a goal or have the "navigation on" checkbox enabled in order
     to activate the teleoperation via controller mode. This has been decided for safety purposes. 
     So if you think that pressing the buttons on the controller doesn't work, first check the aforementioned condition.
     
   2) compile and run the actual program with the launcher file
     $ rosmake tekniker_telepresence
     $ roslaunch tekniker_telepresence tekniker_telepresence.launch 


Known Issues
------------

OpenCV2 has to be compiled with v4l and ffmp support to enable webcam capture. 
You will have to this if you get the "ERROR: webcam capture is NULL" message and no other camera index works. 
Try enabling the v4l and ffmpg flags in opencv2 Makefile and compiling it again
(rosmake opencv2) you may have to delete the ROS_NOBUILD.

As mentioned earlier, if you can't pair the PS3 controller you may have to disable other bluetooth devices for it to work.
http://www.ros.org/wiki/ps3joy/KarmicInstructions

The battery progress bar doesn't change. Our segway driver had an issue and didn't expose the status information.
The callback is done, normalizing the values to a scale and updating the progressbar should be trivial

Streaming kinect audio over ROS doesn't work currently. Libfreenect unstable branch (https://github.com/OpenKinect/libfreenect/tree/unstable) 
has some working demos for the kinects 4 mics, but a ROS node for it is missing.
The easyest way to bypass this hurdle is to add a external mic and use the audio_capture stack

When exiting the application on the client, only the tekniker_telepresence node is finished, the audio_capture and ps3joy nodes have to be terminated with ctrl-c.
Since they are independent nodes, the behaviour is correct, but it would be nice to sync the exiting.

If move_base_server of the navigation system is not activated, the application freezes when trying to navigate because it gets stuck waiting for the server to come up

The path of the controller icon is an absolute path, so you want to change it (look for it at the beginning of ui.cpp)


Future work
-----------

Solving the known issues

Show a map to locate the robot

Give some kind of feedback of the nearest obstacles distance in order to help navigation

Show video and audio stream quality and feedback on the UI like in skype videoconferences

Migrate from wxWidgets to Qt

Because of acumulated errors during the navigation, the  estimation of the (robots) localization could differ from reality. Added this to the uncertainty of the kinects depth sensor, 
it can happen that the robot thinks that the clicked (x,y) destination is unreachable. In the tests, it sometime happened that if the clicked destination (a person) was near wall, the robot couldn't reach the destination because it calculated that the person was inside the wall due to the acumulated errors. An option could be to analyze the surrounding points of (x,y) in a map from map_server and determine if the point is really unreachable. If yes, a nearby destination could be chosen instead