endpointcorp / lg_ros_nodes

A ROS software stack for running Liquid Galaxy applications

Liquid Galaxy

This repo contains ROS software for running Liquid Galaxy applications.

It allows for displaying engaging content in an immersive environment. Currently it supports:

• Google Earth with KML support
• Streetview
• mplayer
• Chrome and...
• every asset that is supported by above apps

General requirements

NOTE: all requirements specific to ros_nodes are in their respective README.md files

• Ubuntu 14.04 LTS
• ros-melodic
• awesome window manager on the top of Xorg for automatic window positioning
• only Nvidia hardware was tested but it should be running with whatever decent accelerated graphics card
• possibility to modify udev configuration
• spacenavigator
• industry standard touchscreen like elo touch 2201L is also a good input device
• it's also good to manage your stack with chef

Running LG

• Pull docker image docker pull endpoint/lg_ros_nodes

• On linux docker run -i -t lg_ros_nodes /bin/bash

• On OSX

• in Xquartz settings go to Security and check Authenticate connections checkbox

• get your IP and set $ip var ip=$(ifconfig en0 | grep inet | awk '$1=="inet" {print $2}')

• allow connections to your Xquartz X server xhost + $ip

• run it! docker run -i -t lg_ros_nodes /bin/bash

Making it running old-fashioned way

First, clone the repos (you can replace ~/src if you want).

$ cd ~/src
$ git clone git://github.com/EndPointCorp/lg_ros_nodes.git
$ git clone git://github.com/EndPointCorp/appctl.git

Then run the init script.

$ cd ~/src/lg_ros_nodes/
$ ./scripts/init_workspace

Then re-run the init script with arguments to direct the script to those new repos

$ cd ~/src/lg_ros_nodes
$ ./scripts/init_workspace --appctl ~/src/appctl/appctl

Install system dependencies with rosdep.

$ cd ~/src/lg_ros_nodes/catkin
$ rosdep install --from-paths src --ignore-src --rosdistro melodic -y

Build the project.

$ cd ~/src/lg_ros_nodes/catkin
$ catkin_make

Run Google Earth, Streetview and Panoviewer sample .launch file:

roslaunch --screen lg_common/launch/dev.launch

NOTE: As new ros nodes are added to this git repo, re-run ./scripts/init_workspace and the rosdep install command.

Development

If you have Liquid Galaxy hardware (headnode + displaynodes), you may want to do the development on it. There's a sync script that will automatically build and transfer your artifact to display nodes and run it afterwards.

To use it:

• make sure that your local catkin/src/ has all nodes that are configured to run on your dispnodes (in their respective roslaunch xml files) - if there are any nodes lacking, they will be ran from /opt/ros directory instead of /home/lg/catkin_ws/ directory where your development build is going to be copied to

• run sync script (do it everytime you want to test sth):

./scripts/sync_to_disp_nodes.sh

This script will build ROS nodes from your locally checked out branch, transfer it to dispnodes and run it by restarting roslaunch service, thanks to the fact that roslaunch service on dispnodes is configured in such way that it attempts to run any development artifact (in /home/lg/catkin_ws) that it finds before launching production ROS nodes that are located under /opt/ros.

Making new release

• newly created ROS nodes need to be listed in the pack-debs script for CI server to build them and distribute to the aptly repository, after CI build is finished, check that the debs exist in aptly under an URL corresponding to the release tag

• release from the master branch

• to make new release you need to:

touch catkin/CATKIN_IGNORE
touch docker_nodes/CATKIN_IGNORE

• make sure your tags are pulled - otherwise you'll generate a duplicate

git remote update

• generate changelog:

catkin_generate_changelog

• edit all your .rst changelogs - remove unwanted or bogus messages and make them look pretty. Use catkin_generate_changelog --all to create CHANGELOG.rst for a new package.

• commit it to prepare for creating new release artifact:

git commit -am "updated changelogs for new release"

• once that's done, prepare release and send it to the build farm:

catkin_prepare_release

NOTE, catkin_prepare_relase ight be a bit broken, so patch /opt/ros/melodic/lib/python2.7/dist-packages/catkin/package_version.py with the below

@@ -127,8 +127,8 @@ def update_changelog_sections(changelogs, new_version):
         new_changelog_data[changelog_path] = data
 
     for changelog_path, data in new_changelog_data.items():
-        with open(changelog_path, 'w') as f:
-            f.write(data)
+        with open(changelog_path, 'wb') as f:
+            f.write(data.encode('utf-8'))
 
 
 def rename_section(data, old_label, new_label):

• remove the CATKIN_IGNORE file, otherwise e.g. catkin_make will ignore packges to compile ...

Global parameters

• /logging/level: sets the minimum log level. Valid values are from 0 to 3: INFO = 0, WARNING = 1, LOG_ERROR = 2, LOG_FATAL = 3. This is not rospy log level but an argument passed to applications managed by appctl

lg_ros_nodes's Issues

Debug socket communication methods for ManagedBrowser

It would be cool to have a method for sending things to the debug socket of a Chrome browser via ManagedBrowser. This implies that a random port should be used unless specified in ManagedBrowser args.

Zombie Earth process

On one occasion when running dev.launch there was a "defunct" Z googleearth-bin process. There was no visible window and it did not respawn.

"Chrome didnt shut down correctly" after roslaunch restart

The --user-data-dir argument points to a directory that needs to be removed each time the ManagedBrowser is launched.

Earth query interface

We need a ROS node for writing Earth queries to the queryfile. There should be one instance of this node running on the same host as the Earth ViewSync master.

Parameters:

• ~query_file [string] - Path to the Earth query file. Default: /tmp/ge_queryfile

Subscribed Topics:

• /earth/query/flyto_kml [std_msgs/String] - A KML AbstractView to fly to.
• /earth/query/flyto_pose_camera [geometry_msgs/Pose] - A Pose expressed in latitude, longitude, degrees, and meters ASL. A <Camera> view will be generated with the absolute Pose values.
• /earth/query/flyto_pose_lookat [geometry_msgs/Pose] - A Pose expressed in latitude, longitude, degrees, and meters relative to sea floor. A <LookAt> view will be generated, with heading, tilt, and roll based on the Pose orientation.
• /earth/query/search [std_msgs/String] - Search string.
• /earth/query/tour [std_msgs/String] - Play a tour by its id. An empty string will exittour.
• /earth/query/planet [std_msgs/String] - Change planets.

Hiller on-site work

List of TODOs:

• add search button
• make the edges of the keyboard round
• keypad relaunch needs to be there
• focus on search form all the time so user doesnt have to touch it to enter search query
• touching the touchscreen moves GE (confirmed)

Need to converge window upon application respawn

This is needed to fix window positioning after a process crashes. Unlike awesome, xdotool will not set a rule.

ManagedApplication has a method for converging its window when its ProcController does a respawn, but need hooks up from ProcRunner to get the notification.

Touchscreen deployments and management

Touchscreen deployments and management https://github.com/EndPointCorp/lg_chef/issues/301 - ticket created for estimation and measurment purposes

Lint config

Initial discussion

https://docs.google.com/document/d/1ATdBJFLRdFeUj1lbnd-gfxBRKfntoDiDDDeQnii4xCw/edit

kmlsync requirement

• needs to unload kmls if no earth windows are in the list

Currently we were ignoring those requests, assuming they were not meant for us

Clean director dependencies:

From matt's comments there are following options:

• Add some kind requirements.txt installing hook in every place that we are deploying (dev, ci, production iso). This makes the package dirty for open-source distribution, which I'm not sure we planned on anyway, but we need to prevent open-source packages from using requirements.txt because it's not kosher with ROS.
• Package the version of Pulsar we want for Ubuntu and add it to rosdep. We can host this dep and additional rosdep source lists locally/privately, too.
• Port from Pulsar to something else. I'm not aware of any specific technical requirement to use Pulsar, but this would not be a simple port.

Pano viewer server

The panoviewer server is a ROS node that mostly just serves as the SpaceNav "driver" for the application. It's pretty similar to the Street View server and may be able to share some code.

• Pub/Sub on /panoviewer/pov with Quaternion message type.
• Pub/Sub on /panoviewer/pano with String message type. This will be a url to the cube map sources.
• Subscribe to /spacenav/twist and publish pov changes.
• Subscribe to /panoviewer/state and disable SpaceNav input when not active.

Support for offline content for Pano Viewer

Related to #27

• support for offline content
• support for link transitions

Bring back Squid caching on roscoe ISO

We disabled squid caching of Maps/Earth data for Portal, which was the basis for the roscoe ISO. We will probably want to set it up again. It deserves some attention so that we might eliminate some legacy squid-related bugs.

360 youtube video

There's going to be an event on 31st of August in Singapore that will need to have support for loading Youtube 360 vids.

According to the burndown chart for MVP we will need to put 360 video feature before MVP milestone is finished.

Adhoc video service server

Consider browser video player

kmlsync ros_node

Requirements:

• it needs to listen for messages from director about KMLs that should be served to GE clients (and therefore keep the state of asset : viewport pair)
• it needs to provide a per-viewport URL to KML for GE
• it needs to know what is the asset that should be displayed on specific GE client (if GE pings KMLsync with different asset that the last one that came from the director, KMLsync should answer with NetworkLinkUpdate KML object)
• it needs to detect tours sent by director and make additional autoplay (query.txt) requests for each tour scene

Development notes:

• implement the ROS equivalent of the KMLMasterWebHandler, KMLUpdateWebHandler, and KMLQueryWebHandler
• older lg-live-image version of the same functionality
• director example message:

{
        "description": "bogus",
        "duration": 0,
        "name": "ts browser",
        "resource_uri": "bogus",
        "slug": "touchscreen-browser",
        "windows": [
        {
            "activity": "browser",
            "assets": [
            "http://lg-head:8060"
            ],
            "height": 1080,
            "presentation_viewport": "touchscreen",
            "width": 1920,
            "x_coord": 0,
            "y_coord": 0
        }
        ]
}

Point cloud viewer

We need a point cloud viewer one day, and now's probably a good time.

https://github.com/eggyknap/lg-potree-2 is a start, but needs a master service to track absolute position, handle director messages, and possibly send out other regular messages with the current point cloud, or other state stuff.

Listen for Panos and 360 vids from the director

Make PV server support Panos from the director.

Requirements:

• in server.py, subscribe to /director/scene
• Message containing activity type of pano should publish a link to panorama on /panoviewer/panoid (this will lead to multiple identical messages but there's really nice way to ignore those)

Side note: CMS message types are defined in the database so we're not tied to any activity_name.

Adhoc browser service

Disable Earth 10 year anniversary splash

There's a 10 year anniversary splash when Earth is launched. Need to find a config key that disables it. Try launching it outside of lg_earth, selecting "Don't show this again", and diffing the config files.

Create (or re-use existing) tool for building debs

Requirements for the tool:

• make it possible to build the debian packages out of a single ROS node
• make it possible to plug it to jenkins

Adhoc video service browser launcher

browser launcher for #6

Mplayer service

kmlsync: use long polling for NLC updates

The original sync_nlc.php had a way of using long polling to help updates apply at nearly the same time across the screens.

Right now kmlsync is doing:

• Get a request for updates.
• Send the response.

This causes content changes to be applied inconsistently across the screens, with difference of up to (and over) a second due to Earth's laziness about timing the requests.

What it should be doing to improve this:

• Get a request for updates.
• If there are updates, send the update response.
• Otherwise, wait up to 10 seconds.
• If an update is needed during those 10 seconds, send the update response.
• If the 10 seconds are elapsed, send an empty response.

lg-ros continuous integration

connect lg-ros to jenkins (lg-ci-01.endpoint.com):

• configure master, development, staging branches
• run tests on each push to those branches with catkin_make run_tests
• throttle those tests so only one instance of ROS-based test is run at a time on lg-ci-01
• configure pull request builder (run tests for all pull requests)

Nagios monitoring for LG-ROS

We need to connect ros_health to nagios

Window management broken on roscoe ISO

xdotool isn't doing what it's supposed to do. ManagedWindow needs to work.

Director deployment changes

Mirrored the ticket from https://github.com/EndPointCorp/lg_chef/issues/300 for estimation and burndown chart purposes

Implement touchscreen autoreconnection

It may happen that director ros node will restart. It provides a websocket channel that browser connects to. Browser needs to automatically reconnect to the director when it goes down.

Investigate alternate approach for running applications

It's possible that we can use a different approach for launching, respawning, and tracking the desired state of applications.

• We can use any blocking call to keep a node alive. This means we can use something as simple as subprocess.call() to run the process. If the process dies, the node dies. Roslaunch can handle respawning.
• Persistent global parameters can be used to track the desired state (think window visibility) of an application, allowing respawns to always return to that state.

What are the benefits?

• No more Thread-based process running.
• Leverage roslaunch respawn feature.
• Less moving parts and state managed in each node.

What are the risks?

• No more low-level process health checking routine. We can implement a new blocking ProcController interface that runs a monitor thread to kill unresponsive processes.
• Node can't change parameters while respawning. Any message changing the desired state must be accompanied by a parameter write. This is complicated by the lack of parameter support in rosbridge. This could be solved by running a separate node just to write the params, or by using latched topics (comes with its own dangers).

Scene handling

dependency on portal-ros

do we want to keep this dependency or port stuff from portal
I am in favour of the latter.

lg_ros deployment + tools #organizational

Full deployment consists of:

• cms - managed by supervisord
• director - change needed - managed by supervisord - should be managed by roslaunch service on the headnode Endpointcorp/lg_chef#300
• touchscreen deployment initial discussion for ispaces ticket is here: Endpointcorp/lg_chef#301 - needs some requirements specification
• ISO ticket - #25
• lg-relaunch - hard to say if we really want one - sudo restart roscore should do the trick
• monitoring - factor out portal_health from portal and make it more generic Endpointcorp/lg_chef#302

this ticket should be divided into subtickets and should not be estimated

Adhoc video service webapp

webapp for server in issue: #6

README instructions

• I think the instructions should be made general, independent on Matt's ~/scr/ locations
• the final command is probably roslaunch lg_common/launch/dev.launch no?

also, since my portal-ros already lives under portal-chrome, I prefer to reference it from there.

if agreed, I can elaborate a more general version. Would like to first hear @mvollrath opinion.

The instructions were otherwise very useful and were working for me - 3 earth windows started, not sure if there is supposed to be any synchronization - moving either of them resulted in no sync in the others.

Streetview server

Support for streetview

Pano Viewer browser launcher

Need a script using ManagedBrowser to launch the Pano Viewer webapp.

• Point at the webapp index in /lg_panoviewer/webapps/client/.
• Pass along rospy params yaw_offset and pitch_offset as url parameters for yawOffset, and pitchOffset as described in #28
• Subscribe to an ApplicationState topic at /panoviewer/state
• Set up the window via viewport param.

Ad hoc media service

Need a ROS node that spins up audio/video players on demand. The behavior should be similar to #5, except launching mplayer instances.

Listen on /media_service/<viewport_name>

Message type AdhocMedia:

• string id
• lg_common/WindowGeometry geometry
• string url

Message type AdhocMedias:

• AdhocMedia medias[]

If the message has no AdhocMedia messages in the array, close everything down.

If the message has AdhocMedia messages in the array:

• Check for an existing mplayer with the same id. If it exists, update the url and geometry.
• If no existing mplayer, create one.
• If an existing mplayer has an id that doesn't match the new message, shut it down and remove it from the tracked instances.

Street View webapp

The browser side of Street View should be pretty thin.

• Create the webapp in lg_sv/webapps/client.
• Set up a Street View scene with the Maps API. https://developers.google.com/maps/documentation/javascript/reference
• Disable all UI elements except the link navigation if url query has showLinks=true.
• Set yaw, pitch and roll offsets from url query yawOffset=N, pitchOffset=N. N will be indices, i.e. 1.0, 2.0 for screens up/right; assume that all windows are the same size for yaw/pitch offset calculations. This will allow for zoom changes independent of a fixed fov.
• When something comes in on /streetview/pov, apply offsets and change the pov to it. To achieve roll, resize and rotate the canvas.
• When something comes in on /streetview/panoid, change the pano to it and publish the metadata to /streetview/metadata.

Pano Viewer webapp

The browser side of pano viewer. Mostly complete with various sync transports, we're looking for a thin version with messaging.

• Create the webapp in lg_panoviewer/webapps/client/.
• Use query string yawOffset and pitchOffset for view offsets. These should be indices like #23
• Listen on /panoviewer/pov for pov changes. The view will be a Quaternion.
• Listen on /panoviewer/pano for pano url changes.

Global paths for common private params

Right now there's a lot of redundant configuration, for example:

<node name="earth_left_one" pkg="lg_earth" type="client" clear_params="true" respawn="true" respawn_delay="3">
  <param name="viewport" value="left_one"/>
  <param name="viewsync_recv" value="true"/>
  <param name="viewsync_hostname" value="10.42.42.255"/>
  <param name="horiz_fov" value="36"/>
  <param name="yaw_offset" value="36"/>
</node>
<node name="earth_center" pkg="lg_earth" type="client" clear_params="true" respawn="true" respawn_delay="3">
  <param name="viewport" value="center"/>
  <param name="viewsync_send" value="true"/>
  <param name="viewsync_hostname" value="10.42.42.255"/>
  <param name="spacenav_device" value="/dev/input/spacenavigator"/>
  <param name="horiz_fov" value="36"/>
</node>

We can factor out viewsync_hostname and horiz_fov (and some others). For example, a global /earth/horiz_fov would serve as the default in absence of an explicit private horiz_fov. This would be set in the head node's ros_params (a new feature of the lg_ros cookbook).

The logic for resolving these params:

if private parameter is set:
  use private parameter
elif global parameter is set:
  use global parameter
else:
  use baked-in default

This will help to reduce configuration bloat.

lg_ros ISO

General requirement: Create another lg-live-image branch for lg-ros that will have similar functionality to portal with some modifications and different deb ROS packages.

We need an lg_ros ISO with following requirements:

• ISO -> everything from portal but:
  • no mosaic (persona var to disable/enable)
  • different metapackage to install them all e.g. ros-indigo-lg-{master,development,staging}
• CI -> this is a good thing to start with: http://wiki.ros.org/buildfarm

Let's stick to jenkins deb builds and aptly repo for now. We will use jenkins builds after everything is opensourced.

child of: #26

Static browser service

• browser service that points to a URL (e.g. portal) - no window management
• expected to be always on (like SV)

SpaceNav emulation

Since spacenavd prevents Earth from reading SpaceNav events, we need a uinput-based solution for writing events out of the ROS nav driver back to a virtual device that Earth can attach to. This has the added advantage of letting us disable the nav input to Earth while it is hidden. Should be a pretty straightforward python-evdev script.

This is required for Earth to co-exist with the ROS driver for other applications.

Generic browser library

• generic browser package that all other browser scripts will use

Touchscreen part I

Requirements:

• POI should come from the director - they're defined as presentations
• no communication with IS master - we dont want any - we dont want to implement deactivate/activate methods for now but this is near future. All angular services (e.g. earth ) are using IS under the hood - we should re-implement them to use director/ROS
• activation/deactivation should come from director - no talking to ROS for now - we want touchscreen to be able to communicate with ROS on "state" topics
• queryfile - we need implement "search" method in "earth" service (mentioned above) to be implemented for now - albatros is a good solution
• implement options like:

{ 
  "show_map": false,
  "allow_search": true,
  "show_relaunch_button": true
}

should be implemented via ROS API - options should be rosparams
For Hiller we dont want touchscreen to follow the map but we want the search button.

Development notes:

• remove IS communication layer with master
• replace IS messaging with ROS messaging - implement rosbridge communication
• implement options

Street View browser launcher

Need a script using ManagedBrowser to launch the Street View webapp.

• Point at the webapp index in /lg_sv/webapps/client/.
• Pass along rospy params yaw_offset, pitch_offset, and show_links as url parameters for yawOffset, pitchOffset, and showLinks as described in #23
• Subscribe to an ApplicationState topic at /streetview/state
• Set up the window via viewport param.

Figure out a good way to do configure viewports and configure nodes to use viewports

We can probably use global params for this. Pretty easy to tack on after launchers have been written.

Touchscreen part II

Here's what's finished: #42

Here's what needs to be done:

• new view for Hiller (need to confirm that)
• touchscreen map should follow earth
• "SV" and "GE" icons should be hidden using TouchscreenOptions parametrization