open-horizon / examples

Code examples you can use with Horizon.

Getting Started

Documentation

Open Horizon documentation repository coming soon! For the time being, you can learn more about Open Horizon here.

Management Hub Installation

Before you can publish and use any of the services in this repository, you must first deploy your own Horizon Management Hub. This can be done with one simple command using the deploy-mgmt-hub.sh script located in the devops repository. This will give you a management hub with several services, policies and patterns published in the exchange.

Register an Edge Node with your Mangement Hub

In order to deploy a service to an edge node it must first be registered with a management hub. The agent-install.sh script is a fast and easy way to register an edge node with a management hub. More information can be found in the open-horizon/anax repository. Edge nodes can be either a device or a cluster. Open Horizon edge cluster capability helps you manage and deploy workloads from a management hub cluster to remote instances of OpenShift® Container Platform or other Kubernetes-based clusters.

Typically, edge devices have a prescriptive purpose, provide (often limited) compute capabilities, and are located near or at the data source. Currently supported edge device OS and architectures:

• x86_64
  • Linux x86_64 devices or virtual machines that run Ubuntu 20.x (focal), Ubuntu 18.x (bionic), Debian 10 (buster), Debian 9 (stretch)
  • Red Hat Enterprise Linux® 8.x and 9.x
  • Fedora Workstation 32 - 37
  • CentOS 8.x
  • SuSE 15 SP2
• ppc64le (support starting Horizon version 2.28)
  • Red Hat Enterprise Linux® 7.9
• ARM (32-bit)
  • Linux on ARM (32-bit), for example Raspberry Pi, running Raspberry Pi OS buster or stretch
• ARM (64-bit)
  • Linux on ARM (64-bit), for example NVIDIA Jetson Nano, TX1, or TX2, running Ubuntu 18.x (bionic) or Raspberry Pi OS 64bit
• Mac
  • macOS (Intel)

Open Horizon edge cluster capability helps you manage and deploy workloads from a management hub cluster to remote instances of OpenShift® Container Platform or other Kubernetes-based clusters. Edge clusters are edge nodes that are Kubernetes clusters. An edge cluster enables use cases at the edge, which require colocation of compute with business operations, or that require more scalability, availability, and compute capability than what can be supported by an edge device. Further, it is not uncommon for edge clusters to provide application services that are needed to support services running on edge devices due to their close proximity to edge devices. Open Horizon deploys edge services to an edge cluster, via a Kubernetes operator, enabling the same autonomous deployment mechanisms used with edge devices. The full power of Kubernetes as a container management platform is available for edge services that are deployed by Open Horizon. Currently supported edge cluster architectures:

• OCP on x86_64 platforms
• K3s - Lightweight Kubernetes
• MicroK8s on Ubuntu 18.04 (for development and test, not recommended for production)

Currently there is only one example service in this repository that is designed to run on an edge cluster and that is the nginx-operator.

Example Services

During the management hub installation, several services should have been published into the exchange automatically. The following three command will list the services, patterns, and deployment policies available in your exchange:

hzn exchange service list IBM/
hzn exchange pattern list IBM/
hzn exchange deployment listpolicy

Note: The above commands assume you have the Horizon environment variables HZN_ORG_ID and HZN_EXCHANGE_USER_AUTH set.

You can find a list of available edge services in this repository located in the edge/services directory. For the most part, each of the services are broken up into micro-services designed to accomplish one specific task. This makes them easier to incorporate into a wide variety of "top-level" services.

A good example of a "top-level" service is cpu2evtstreams, which has two dependent services (cpu_percent, and gps). It uses these two micro-services to gather information about the edge node it is running on and sends it to an instance of IBM Event Streams using kafkacat.

Edge examples specific to the Watson IoT Platform are found in edge/wiotp. These examples are not being maintained.

The backend infrastructure for the sdr2evtstreams service has been deprecated and removed from this repo. If you wish to access the most recent version, you can find it in the deprecated_cloud_sdr_example branch or any open-horizon/examples release v2.30.0-934 or older.

Using Example Services

Each example service in this repo has a README that includes steps to run it when it is currently published in your exchange, or a "Create your own" set of instructions that will guide you through the process of publishing your own version to your exchange.

Contributors

examples's Issues

Restructure new samples and move to open-horizon/examples

• Get Final list of "approved" samples to move to open-horizon/examples (Dave Booz now says he isn't sure if he wants to support the --net=host option)
• Get USB audio adaptor sound card to test Pooja and Lily's samples

Wifi Signal Scan sample - Troy

• Requires --net=host
• Services to Restructure
  • wifisignalscan
  • wifi2msghub
  • gps

Offline Voice Assistant - Pooja

• Services to Restructure
  • processtext
  • text2speech
  • audio2text
  • mqtt_broker
  • voice2audio
• Tested and Registered on ICP major-peacock exchange
• Write README and service/pattern description
• PR Submitted

Smart Audio w/Edge - Lily

• Services to Restructure
  • watson_speech2text
  • hotword_detection (wrote new asound.conf file to work for my usb sound card)
  • stopword_removal (rewrote service as WSGI server)
  • mqtt (changed topic names to more generic names to understand where the communication is coming from and going to)
  • mqtt2kafka (works on ICP)
• Added Makefile's for all of the above services
• Tested and Registered on ICP major-peacock exchange
• Write README and service/pattern description
• PR Submitted

Security Camera Sample - Sadiyah

• Services to Restructure

Edge Gateway - Sanjeev

• Requires --net=host
• Services to Restructure
  • efg (Edge Fabric Gateway)
  • ble
  • mqtt_broker
  • mqtt_kafka
  • oled

Noise Compliance through Situational Awareness - James

• Services to Restructure

Edge Fabric Chatbot - Ivan

• Services to Restructure

Intranet Performance Monitor - Nadim

• Requires --net=host
• Services to Restructure

Switch2Msghub - Abhishek

• Requires --net=host
• Services to Restructure

Network Observer - Max

• Requires --net=host
• Services to Restructure

(SDR UI) Firefox ESR (60.3.0esr) (64-bit) Causes Network Error

The app displays a network error followed by an automatic re-direct back to the login page.

Create edge service packaging format to enable copying services

There is a poor man's example of copying services in https://raw.githubusercontent.com/open-horizon/examples/master/tools/loadResources . But a better approach is for us to define a packaging format for service definitions (and associated patterns and deployment policies):

• tar format that holds the horizon dir
• when we build the pkg, we push the built images to docker hub and include a reference in the pkg. Or we docker save the images to tar files and include them in the pkg.
• add list of arches to horizon dir?
• add post-publish script?
• hzn sub-cmd to publish 1 service-pkg or a tar file of them:
  • 1st implement in script (both build pkg and publish pkg), then convert to hzn sub-cmd
  • unpacks and runs: hzn exchange service publish -O -P -f horizon/service.definition.json
  • also optionally publishes: pattern, dep pol, service pol
• hzn sub-cmd to get a service from the exchange and package it
• The enables a user to copy services from 1 org to another, or 1 exchange to another

As a reference, there is a sample telco pkging format: https://www.etsi.org/deliver/etsi_gs/NFV-SOL/001_099/004/02.07.01_60/gs_NFV-SOL004v020701p.pdf

Clean up rest of open-horizon/examples

• Remove wiotp/location2wiotp. Remove wiotp/cpu2wiotp?
• Change cpu to be structured like helloworld
• Change gps to be structured like helloworld
• Change sdr2msghub to be structured like helloworld
• Change sdr to be structured like helloworld
• In service.definition.json move env vars from deployment.services.*.environment to userInput
• Change lower level services to have their rest api on port 80
• Create a documentation page for each sample, describing what it does and its API (if it has one). Have its documentation field point to it.
• Add and test cross-arch building to sdr and sdr2msghub Makefiles
  • sdr
  • sdr2msghub
    • amd64
    • arm
    • arm64
• Create updated service and pattern definitions for all of the above samples in both staging and prod

Test in all 3 modes the examples are used in:

• IBM Edge Fabric Developer building/publishing the example: use the default variables defined in horizon/hzn.env, build/hzn dev/publish/hzn register for all 3 arches and use a pattern that includes all 3.
• Customer Service Developer: following dev guide, change variables in horizon/hzn.env, build/hzn dev/publish/hzn register for current machine's arch.
• Customer User: following "Registering Your Edge Machine", hzn register on their edge node.

Add edge node stations page to SDR UI?

On the edge node top nouns page, add a link to a new page: top stations, which will show a table of fields: frequency, numberofclips, dataqualitymetric, timeupdated from the stations table, ordered by: numberofclips desc, timeupdated desc limit 20.

Note that a frequency of 0.0 indicates that audio is coming from the internet, so the UI should replace 0.0 with (internet) in the table.

Also note that soon we will be adding column callletters to the stations table. In the case of an "internet station" the callletters field might contain the url.

Part of epic https://github.ibm.com/Edge-Fabric/project-summary/issues/35

Fix/workaround SDR hardware lockup

Currently, (I think only on RPi?) if rtl_* is called every 30 seconds for some hours / days, the hardware will stop working. There are two solutions:

• Reset the USB bus whenever the SDR gets stuck. This is ugly.
• Use rtl_rpcd. The code base has not been maintained for 4 years.
  Both of these have issues.
• Decide which is less bad
• Implement the chosen fix
• Test for several days on my three RPis to gain some evidence that the problem is actually gone

Fix SDR node vulnerabilities as of 4/1/2019

Vulnerable packages:

• extend (https://github.com/open-horizon/examples/blob/master/cloud/sdr/ui/sdr-app/client/package-lock.json)
• extend (https://github.com/open-horizon/examples/blob/master/cloud/sdr/ui/sdr-app/package-lock.json)
• lodash (https://github.com/open-horizon/examples/blob/master/cloud/sdr/data-processing/ibm-functions/actions/package-lock.json)
• lodash (https://github.com/open-horizon/examples/blob/master/cloud/sdr/ui/sdr-app/package-lock.json)
• lodash (https://github.com/open-horizon/examples/blob/master/cloud/sdr/ui/sdr-app/client/package-lock.json)
• webpack-dev-server (https://github.com/open-horizon/examples/blob/master/cloud/sdr/ui/sdr-app/client/package-lock.json)
• react-dom (https://github.com/open-horizon/examples/blob/master/cloud/sdr/ui/sdr-app/package-lock.json)
• react-dom (https://github.com/open-horizon/examples/blob/master/cloud/sdr/ui/sdr-app/client/package-lock.json)
• react-dev-utils (https://github.com/open-horizon/examples/blob/master/cloud/sdr/ui/sdr-app/client/package-lock.json)

Encourage use of service urls that do not require exchange munging

The Problem: After a service is created, hzn exchange service list shows them like IBM/github.com-open-horizon-examples-wiki-service-cpu2msghub_1.2.5_amd64. The url value has to be munged because a regular url can't be passed into the exchange rest methods as part of the service id, because it would mess up the route of the rest method. This presents a usability problem because in other places in which a user needs to specify a service url (e.g. other services or pattern), they might think they should use github.com-open-horizon-examples-wiki-service-cpu2msghub. Also, in the 2+ years we've had service/microservice/workload definitions in the exchange, i've never gotten any traction (even with the horizon team) on using the url field as the link to the documentation (which was part of the original reason for using the url as part of the unique identifier. Because of this, users have been confused about why it is a url and what they should set it to. Note that since the introduction of orgs, service identifiers only need to be unique within the org, not globally.

Solution Options:

1. Drop the desire to have the url field double as a pointer to documentation, and encourage service developers to use a url value that won't have to be munged, e.g. github.com.open-horizon.examples.cpu2msghub. This is similar to how java packages are named. Note that this approach works today. The only thing we might want to add is a warning or error if the url value contains chars that require munging (but we will wait on that). Also add an optional new field in the service definition called documentation that explicitly points to the service documentation.
2. Have the hzn command completely hide the munged url-version-arch. So only exchange rest api exploiters would ever need to deal with it. Hzn would always refer to services with the regular url, version, and arch for both creation and view (but under the covers still munge it). This will be problematic because some of the error msgs (about a referenced service not existing come from the exchange.
3. Change the service resource in the exchange such that it is given an auto-generated id like 123059851202345981043958123. Handle the migration issues, and make hzn changes such that the user doesn't have to keep doing the mental mapping from id to url, version, and arch.

We are planning to do option 1.

The functionality of the bbcfake portion in the SDR no longer working

It seems the bbc website has changed in such a way that the scraping functionality of the bbcfake no longer works as intended and is no longer returning URL's properly. This has been verified on a VM edge node in it's syslog with errors that look like this:

Apr 18 21:16:18 ctiijimatest02 workload-ibm_github.com.open-horizon.examples.sdr_0.0.7_4da208c9-0b1c-49fd-b07a-449ced9edd4e_sdr[1043]: got 0 urls
Apr 18 21:16:18 ctiijimatest02 workload-ibm_github.com.open-horizon.examples.sdr_0.0.7_4da208c9-0b1c-49fd-b07a-449ced9edd4e_sdr[1043]: no urls, refreshing
Apr 18 21:16:18 ctiijimatest02 workload-ibm_github.com.open-horizon.examples.sdr_0.0.7_4da208c9-0b1c-49fd-b07a-449ced9edd4e_sdr[1043]: fetching a new set of URLs

This repeats infinitely on the edge nodes syslog.

GPS service returns non-zero elevation when using coordinates from IP address.

The http://freegeoip.net/csv/ api used by the horizon gps service was deprecated, so the ip estimated location stopped working, so i updated that code to use the new api. In the process i noticed that get_location_from_ip_address() doesn’t return elevation, but the gps service still returns it as 114.6 (at least for my location). From looking at the code, i couldn’t figure out how that was happening. Please debug and fix. Thx.

Add variable data mock to SDR PoC

Mock mode for sdr2msghub that gets audio from BBC radio.

• Extract URLs from BBC page
• Decode mp3 and down sample
• Slice audio into correct length chunks.
• For mock data, set the frequency to 0.0
• Add additional logging when VERBOSE=1
• Make it clear in the log when sdr2msghub sends its first audio clip (VERBOSE=0) or when it sends every audio clip (VERBOSE=1). Depending on antenna quality, position, and geographic location, it can take an hour or more for sdr2msghub to send its 1st clip, leaving the node owner wondering if it is working correctly or they did something wrong. We need to give them easily identifiable info in the log when that important event occurs, and if not yet, why not.

Part of https://github.ibm.com/Edge-Fabric/project-summary/issues/8

SDR POC UI completion

• Add multiple page/route support
• Get nav bar working with carbon
• Add mapbox page showing dots for all of the participating edge nodes
• Add page showing 1 edge node's top 20 keywords/sentiments when that edge node is clicked on in the map
• Add title to UI

(Moved Authenticate IBM Cloud users to separate issue in sprint 3.)

Part of https://github.ibm.com/Edge-Fabric/project-summary/issues/4

`hzn dev dependency fetch --url` does not deterministically default arch

Issue reported by Chris trying to run through the dev guide on a TX2 (arm64).

Doing an hzn dev dependency fetch --url on that host for the gps service sets "arch": "amd64" in the service definition JSON file (but this should be arm64):

"requiredServices": [
        {
            "url": "https://$MYDOMAIN/service-$CPU_NAME",
            "org": "$HZN_ORG_ID",
            "version": "$CPU_VERSION",
            "arch": "$ARCH"
        },
        {
            "url": "https://github.com/open-horizon/examples/wiki/service-gps",
            "org": "IBM",
            "version": "2.0.5",
            "arch": "amd64"
        }
    ],

Bruce examined the code and says:

If i’m reading the code correctly (dave wrote that code), if arch isn’t specified it takes the 1st one it comes to (that is at the highest version), which isn’t the correct behavior

Yes, the code which looks up the dependency in the exchange does not default the arch to the arch of the system where the user is running hzn dev commands.

• implement and test

Change example msghub dirs/files, and env vars to evtstreams

• env vars to EVTSTREAMS_*
• cpu2msghub to cpu2evtstreams
• dir msghub to evtstreams
• ask rene to do the corresponding small changes to KC

Investigate occasional long timeout of cpu2msghub on mac

Got a little more info about this problem that we've been seeing:

• Sometimes (depends on timing i think) the 1st time that cpu2msghub runs curl to the cpu service rest api, it gets caught in a state in which the curl cmd takes more than 5 minutes (didn't time it exactly) to return and then returns rc 56. On the next interval (and every time after that) the curl cmd to cpu works fine.
• During this time, i can docker exec into the cpu2msghub container and run the same curl cmd successfully (quickly get 200 http code and real cpu data).
• I added --connect-timeout 5 to the curl cmds, but it didn't help this problem.
• This info leads me to believe that when the timing it just right, curl is connecting to something else on port 80 (before docker has mapped port 80 to the cpu service), which is then never returning a response.
• I've only been able to recreate this on my mac (not a linux edge node). I've seen it for both the amd64 and the arm64 containers.
• I've seen it when run both via hzn dev service start and hzn register ...

We've run cpu2msghub for a long time w/o seeing this problem, so the question is why has this come up now? Is it caused by a new version of docker? New version of anax? Listening on port 80? Some investigation steps:

• try to recreate it thru repetition
• try switching the cpu service to listen on a port other than 80
• add -v to the curl cmds to get more info on who it connected to
• google, google, google
• if all else fails, a sleep in cpu2msghub might work around this problem, but this is a last resort. You never know who much of a sleep is needed on all platforms. Plus, part of our best practice doc is to just retry until required services respond.

cpu2msghub & helloworld new structure and doc updated

• Add simple helloworld sample (bruce)
• Add helloworld to the registration page and the dev guide (ivan reviewing)
• Change cpu2msghub to be structured like helloworld (bruce)
• Change cpu2msghub to not use HZN_PATTERN for the msg hub topic
• Modify dev guide to give instructions for starting cpu2msghub (get msghub api key, create topic) after the helloworld sample. Remove the cpu2msghub step-by-step commands that are now in the cpu2msghub README.md (ivan reviewing)

Test in all 3 modes the examples are used in:

• IBM Edge Fabric Developer building/publishing the example: use the default variables defined in horizon/hzn.env, build/hzn dev/publish/hzn register for all 3 arches and use a pattern that includes all 3.
• Customer Service Developer: following dev guide, change variables in horizon/hzn.env, build/hzn dev/publish/hzn register for current machine's arch. (ivan testing)
• Customer User: following "Registering Your Edge Machine", hzn register on their edge node. (ivan testing)

Signal strength mapper sample

• Scan for signal data of all in range WiFi networks (service.sh)
• Create Docker container to run service.sh
• Create Horizon Service to run wifi scan (wifisignalscan) ...may not be possible with --net=host?
• Establish communication via REST API between wifi2msghub and wifisignalscan
• Create top level service (pattern) that will talk to gps and wifisignalscan
• Get location from gps using REST API ("Estimated" at the moment)
• Solve issue of getting scan info out over REST API
• Once data is flowing to wifi2msghub, format it into json structure Jay requested
• Solve issue of second curl failing to wifisignalscan
• Dockerize C wifi2msghub program so I can use Kafkacat and send to an Event Stream on pi
• Send data to Event Stream
• Get data into database in the IBM Cloud using IBM Functions
• Store data into mysql database

SDR UI: left nav item should be highlighted

Items in the left navbar should be highlighted to indicate it is the current route. Currently, it does not reflect the route.

Learn SDR edge service code and take over support for it

• Learn the code for, and be able to build, sdr edge docker image and service definition
• Learn the code for, and be able to build, sdr2msghub edge docker image and service definition
• Learn the code for, and process of, training the radio station detection model

Get GPS PoC to run unprivileged

Currently, the gps sample service sets the deployment.services.gps. privileged field to true so it can access the gps sensor on the edge node. (See

examples/edge/services/gps/horizon/service.definition.json

Line 17 in 3be5c11

"privileged": true,

.) This isn't ideal because services should run with the least privilege possible, because they are running on someone else's edge node.

Instead of using the blanket privileged access, it should be possible to use the cap_add field to get the access needed to the gps sensor. (See https://github.com/open-horizon/anax/blob/master/doc/deployment_string.md#deployment-string-fields .)

This will be a good example to service developers of how to address a common issue. To get details about the cap_add field, talk with ling.

Modify and open src PWS POC

• rebuild/automate the pws ms docker image
• remove any user input vars that can be from either ms or wk (e.g. currently they both need HZN_PWS_MODEL, that could be reduced to 1 of them)
• determine if the correct devices in the deployment field can be used instead of privileged mode
• maybe add purple or AQI info the data

Complete the full Developer Guide so it is ready for general use

there is a lot of good info in the developer guide you wrote that i want to make available to users. I know it needs updating and i’ve been thinking about the best way to make use of the guide and here’s my thoughts: the dev quick start has the step-by-step, but almost nothing giving the bigger picture about how all the pieces fit together and general concepts and architecture about horizon and how it fits with wiotp. The developer guide already has a lot of this, so i suggest transforming it into a Horizon Principles and Concepts doc (title can be changed).

I don’t think it is worth it to try to have step-by-step commands in the developer guide. I don’t want us to have 2 places we have to maintain that info, and with the developer guide being longer, it will always take longer to review

in my mind that also goes for the step-by-step of setting up bluemix, wiotp ui, etc. I know we only cover that very tersely in the quick start, but i’m hoping the wiotp team can provide that info

the updates to the developer guide would also have to include converting it from microservice/workload to services. See the services design for some background info on that: https://docs.google.com/document/d/1xKCPP8Rx_KTGZxYFOl4ZEg-hE1qGO8eUAf-i_FyGzGA/edit#heading=h.sb4u7ixyrfmz

Repository insecure

Package repository (pkg.bluehorizon.network) is only accesible via http. The repository public key is also only accessible via http.

Update cpu2msghub to exploit anax 2.18.1 features

Part of https://github.ibm.com/Edge-Fabric/project-summary/issues/1

Add sample testing script

To simplify the sample Makefiles and avoid specifying a sleep time, which will never be right for all platforms, create a bash script that can be used, and put it in the tools dir. It should have these features:

• read docker logs -f <container> on mac (Darwin)
• read tail -f /var/log/syslog | grep <regex> on linux
  • test on linux (RPi on major-peacock icp)
• take as args either grep args or jq args to search/parse the output
• as soon as the required output is detected, exit with a zero status code
• have a configurable timeout value. If the required output isn't found by that time, then exit with a non-zero status code

Status:
In Makefile

# Configurable parameters for testing service
export MATCH:='echo {"nodeID":'
export TIME_OUT:=60

test:
    hzn dev service start -S
    @echo 'Testing service...'
    ../../../tools/serviceTest.sh $(SERVICE_NAME) $(MATCH) $(TIME_OUT) && \
        { hzn dev service stop; \
        echo "*** Service test succeeded! ***"; } || \
        { hzn dev service stop; \
        echo "*** Service test failed! ***"; }

Service Test Script:

#!/bin/bash

########################### TESTING ############################
# For now on Mac:
# docker logs -f `docker ps -q --filter name=$SERVICE_NAME`
#
# For now on Linux:
# sudo tail -f /var/log/syslog
################################################################

# $1 == $SERVICE_NAME
# $2 == "key" being searched for to know if service is successfully runnning. If found, exit(0)
# $3 == timeout - if exceeded, service failed. exit(1)

name=$1
match=$2
timeOut=$3
START=$SECONDS

##################################### Check the operating system #########################################
if [ $(uname -s) == "Darwin" ]; then
    # This is a MAC machine
    command="docker logs -f `docker ps -q --filter name=$name`"
else
    # This is a LINUX machine
    command="sudo tail -f /var/log/syslog"
fi

####################### Loop until until either MATCH is found or TIMEOUT is exceeded #####################
$command | while read line; do
    # MATCH was found
    if grep -q -m 1 "$match" <<< "$line"; then
        exit 0

    # TIMEOUT was exceeded
    elif [ "$(($SECONDS - $START))" -ge "$timeOut" ]; then
        exit 1
    fi

    sleep 1;

done

Where to ask questions ?

The discourse server (https://discourse.bluehorizon.network/) does not allow any new entries .
How may we ask questions or report problems ?

How to get HZN_ORG_ID

Hi,
I tried to follow "Hello-World" example. In this i am stuck with "hzn exchange user list" command. It returns

Error: bad HTTP code 401 from GET https://alpha.edge-fabric.com/v1/orgs/HZN_ORG_ID/users/iamapikey, output: {"code":"invalid-credentials","msg":"IAM authentication succeeded, but no matching exchange org with a cloud account id was found for <HZN_ORG_ID"}

Here i am setting HZN_ORG_ID as my email address which i have used to create my free IBM Cloud account.
Can you guide my about HZN_ORG_ID, from where can i get it.
Thanks

Add user authentication to SDR POC UI

• Structure the authentication such that there could be plugins for multiple clouds
• Implement an auth plugin for IBM Cloud (including SSO) such that only cloud users in the same account as the SDR UI is running in can view any of the pages of the UI
• Also add graphql subscriptions to have the Global Keywords and Edge Nodes pages update with new data as it is written to the db

The `cloud/sdr/deploy/ibm/deploy.sh` script stops silently (no error) on MacOS

User @dnguyenv reported that when he tried to run deploy.sh on his MacOS machine that it just stopped (no error message) if you don’t have any instance of the services already created in your IBM Cloud account. Apparently it will stop at lines 160 , 189, 212,241,and 264, but if you manually create the services, and comment out the lines listed, then it will run.

Here is some of the chat from a thread in the user channel attached to the message from @dnguyenv at 1:54PM Pacific on Friday Dec 14:

mosquito [12 minutes ago]
@dnguyenv Yes, but could you please share what errors you got when you tried to run it and it stopped at those lines? I’d like to understand what went wrong.

dnguyenv [5 minutes ago]
it just stops there without any error message, let me show you

dnguyenv [4 minutes ago]

2018-12-14 17:11:35 Checking prerequisites:
=============================================================
Found jq
Found curl
Found ibmcloud
Found psql
Found sed
Found grep
Found cut
Found npm
Found node
2018-12-14 17:11:35 All prerequisites are met
=============================================================
2018-12-14 17:11:35 The current action is deploy for all component(s)
2018-12-14 17:11:35 Creating all service instances required for running SDR PoC
2018-12-14 17:11:35 Creating edge-sdr-poc-speech-to-text Watson Speech-To-Text service instance with the standard plan
2018-12-14 17:11:35 Current Watson Speech-To-Text instances:```

cpu2msghub logs syntax error until it can communicate with cpu and gps

The beginning of the cpu2msghub log looks like:

Optional environment variables (or default values): SAMPLE_INTERVAL=2, SAMPLE_SIZE=5, PUBLISH=true, MOCK=false
Checking for required environment variables for publishing to IBM Message Hub:
  HZN_ORGANIZATION=IBM
  HZN_DEVICE_ID=bpmac
  MSGHUB_TOPIC=cpu2msghub
  MSGHUB_API_KEY=LheItOe2b6UquCt7C_hdUEuQdSI1tsOXygr03gwAwyT4
  MSGHUB_BROKER_URL=9.30.210.34:30226
Will publish to topic: cpu2msghub
Starting infinite loop to read from the cpu and gps services then publish...
jq: error (at <stdin>:1): Cannot index number with string "cpu"
 Interval 1 cpu: 
(standard_in) 2: syntax error
 Interval 2 cpu: 1
(standard_in) 1: syntax error
 Interval 3 cpu: 0.25
(standard_in) 1: syntax error
 Interval 4 cpu: 0.75
(standard_in) 1: syntax error
 Interval 5 cpu: 0.75
(standard_in) 1: syntax error
(standard_in) 1: syntax error
echo {"nodeID":"bpmac","cpu":,"gps":{"lat":42.0738,"lon":-73.9797,"alt":50.05}} | kafkacat -P -b 9.30.210.34:30226 -X api.version.request=true -X security.protocol=sasl_ssl -X sasl.mechanisms=PLAIN -X sasl.username=token -X sasl.password=LheItOe2b6UquCt7C_hdUEuQdSI1tsOXygr03gwAwyT4 -X ssl.ca.location=/tmp/es-cert.pem -t cpu2msghub

Also, the 1st msg sent to event streams was:

cpu2msghub/0/3/: {"nodeID":"bpmac","cpu":,"gps":{"lat":42.0738,"lon":-73.9797,"alt":50.05}}

(Note there is no value after the "cpu" field. This is invalid json.)

It is normal/expected that a service might not get a good result from required service api's for the 1st few tries, because all the services start concurrently, so the top-level service might start calling the required services before they have finished initializing. But in this case cpu2msghub should handle those errors more gracefully and not log syntax errors and not send invalid json to event streams.

BTW, i saw these error when running it via hzn dev service start -S. I think they will also happen when run via a full hzn register .... But it will be easier for you to debug with hzn dev service start -S.

Offline voice assistant sample

• All 4 services working on rpi
• All 4 services working independetly in docker
• Link the services using MQTT and REST
• Horizonize the services

Change cpu2msghub required versions back to [0.0.0,INFINITY)

Do this after open-horizon/anax#1041 is fixed:

• Change the version fields in the cpu2msghub requiredServices section back to [0.0.0,INFINITY)

Blocked by open-horizon/anax#1041

Streamline Developer's Quick Start Guide

• modify hzn to always substitute env var values in input files
• modify dev quick start guide to make use of that
• modify the examples services so they can be "imported" simply with cp -a
• exploit new hzn and wiotp ability to create a gw type/pattern with the desired services

Implement service development CI/CD process

And document for customers.

DELETED: Update all other examples to match the `cpu2msghub` changes in issue #140

Make changes similar to those made to cpu2msghub in #140 in all of our other services:
[ ] gps
[ ] cpu
[ ] sdr
[ ] sdr2msghub

Then follow the procedure in https://github.ibm.com/Edge-Fabric/project-mgmt/issues/151 to release these changes.

Add yolo/detect sample to open-horizon/examples

Working with Glen, move the services from https://github.com/TheMosquito/detect into open-horizon/examples.

Need to add instructions to README.md, and we should consider adding a pointer to it in the KC getting started section.

Add station callletters to SDR POC?

Some FM radio stations broadcast their call letters. This is useful to help users connect a frequency with an actual station, and to associate a station that is broadcast on multiple frequencies (usually by repeaters). It can also be useful to identify internet radio "stations".

• Add callletters column to stations table in db, as an additional key. (No automated migration - sdr is an example)
• Modify the sdr service to get the call letters from FM stations that broadcast it
• Modify the sdr service to put the url of internet radio "stations" (used for mocking) in the call letters field
• Modify the sdr2msghub service to add the callLetters field to the json sent to the cloud
• Modify the ibm functions action to put the call letters in the callletters column of the stations table
• Modify the sdr UI to query this value and show it in the edge node page

Part of epic https://github.ibm.com/Edge-Fabric/project-summary/issues/35

Escape quotes in wiotp_agent_setup

Running Ubuntu, I had some troubles to run the wiotp_agent_setup to register my edge on Watson. I had to edit the wiopt_agent_setup file manually to escape some quote characters, otherwise I got this error:
export WIOTP_INSTALL_DEVICE_TOKEN=token
As you can see, the token is not "in quotes", but it should be.

It would be more convenient to correct this in the code, so other users won't have the same problem.

Thanks

Investigate min manual steps to use workload/ms image in bx registry w/horizon

The current doc of how to do this is: https://docs.google.com/document/d/1Qlff9PT_o8Qv83viKKLZda5L-o0QdRm_8Cqi7MpA1qU . In our bluehorizon hybrid environment, this was done for location2wiotp.

The basic use of images stored the bx container registry is documented in https://docs.google.com/document/d/1awe8WZxJbXE1Xt2GiiSrODQfgrhWCTWtUGG0ZWbGcXI . But that is a lot of manual steps required on every edge node using the image. So i'd like to see an end-to-end bulleted list flow with the minimum possible steps to both put the image in the registry, and to get many edge nodes to be able to use it. Including the highest priority updates to horizon to streamline the process.

POC exploiting aural2 in openhorizon/examples

Isaac will put the new microphone ms and a sample wk to use it on github.com/openhorizon/examples. He'll also put the aural2 docker image in docker hub, exch ms def file in openhorizon/examples, and a sample workload to use it

Modify bluehorizon.network web pages to direct users to wiotp

EDIT: See longer post below for more details on this task.

Only change bluehorizon.network. Do not modify staging.bluehorizon.network for now.

• Point to our quick start guide, dev quick start guide, and ultimately the official wiotp edge documentation.
• When wiotp form of our POCs are working and interesting, point those pages to them
• Direct to wiotp forum and https://github.com/open-horizon/examples/issues instead of discourse
• Redirect other pages as appropriate

Create a script that the exchange can run on initial startup to create the samples in the exchange

This script will require:

• ability to run hzn commands
• HZN_EXCHANGE_USER_AUTH credentials to be set
• cloning open-horizon/examples git repo
• new target in Makefiles to publish service or pattern without running 'docker build'

Uses feature open-horizon/anax#967

Currently blocked by issue open-horizon/anax#1084

Define service development CI/CD process

Use the same process for our own samples and what we suggest to our customers.

Design is here: https://ibm.ent.box.com/notes/542507981970

Also investigate gitflow to see if it would be useful.

Add horizon bx cr doc to examples documentation

This doc explains the steps to put a docker image (that is part of a horizon service) into the bx cr registry and use it with horizon: https://docs.google.com/document/d/1Qlff9PT_o8Qv83viKKLZda5L-o0QdRm_8Cqi7MpA1qU

Convert it to markdown, place it in examples/edge/doc, and link to it from the relevant other docs.

GPS container didnt handle error result from ifconfig.co

When running the SDR POC on my RPI, the GPS container failed to estimate my GPS location. The following was in the GPS container log. It appears that ifconfig.co return an error and the GPS code didnt handle it, just kept going and made the call the api.ipstack.com. I have confirmed that the failure from ifconfig.co is not permanent, it worked on my rpi while debugging this issue, so perhaps simply adding an intermitent retry until it succeeds is sufficient to solve the problem.

Oct 18 21:16:56 raspberrypi workload-github.com-open-horizon-examples-wiki-service-gps_2.0.5_ae9f6194-7f1f-42cd-9137-a5b5449f7d25_gps[396]: 2018/10/18 21:16:56 WARNING: Env variable HZN_USE_GPS should have been set, but it was not.
Oct 18 21:16:56 raspberrypi workload-github.com-open-horizon-examples-wiki-service-gps_2.0.5_ae9f6194-7f1f-42cd-9137-a5b5449f7d25_gps[396]: 2018/10/18 21:16:56 Estimating node location using public IP address.
Oct 18 21:17:08 raspberrypi workload-github.com-open-horizon-examples-wiki-service-gps_2.0.5_ae9f6194-7f1f-42cd-9137-a5b5449f7d25_gps[396]: Response from http://api.ipstack.com/Error from ifconfig.co: Get http://ifconfig.co: dial tcp [2606:4700:30::6812:3f80]:80: connect: cannot assign requested address?access_key=166f0d44636d9ed71c6a01530f687fdb&format=1:
Oct 18 21:17:08 raspberrypi workload-github.com-open-horizon-examples-wiki-service-gps_2.0.5_ae9f6194-7f1f-42cd-9137-a5b5449f7d25_gps[396]: {
Oct 18 21:17:08 raspberrypi workload-github.com-open-horizon-examples-wiki-service-gps_2.0.5_ae9f6194-7f1f-42cd-9137-a5b5449f7d25_gps[396]: "ip":"Errorfromifconfig.co:Gethttp:/ifconfig.co:dialtcp[2606:4700:30::6812:3f80]:80:connect:cannotassignrequestedaddress",
Oct 18 21:17:08 raspberrypi workload-github.com-open-horizon-examples-wiki-service-gps_2.0.5_ae9f6194-7f1f-42cd-9137-a5b5449f7d25_gps[396]: "type":null,
Oct 18 21:17:08 raspberrypi workload-github.com-open-horizon-examples-wiki-service-gps_2.0.5_ae9f6194-7f1f-42cd-9137-a5b5449f7d25_gps[396]: "continent_code":null,
Oct 18 21:17:08 raspberrypi workload-github.com-open-horizon-examples-wiki-service-gps_2.0.5_ae9f6194-7f1f-42cd-9137-a5b5449f7d25_gps[396]: "continent_name":null,
Oct 18 21:17:08 raspberrypi workload-github.com-open-horizon-examples-wiki-service-gps_2.0.5_ae9f6194-7f1f-42cd-9137-a5b5449f7d25_gps[396]: "country_code":null,
Oct 18 21:17:08 raspberrypi workload-github.com-open-horizon-examples-wiki-service-gps_2.0.5_ae9f6194-7f1f-42cd-9137-a5b5449f7d25_gps[396]: "country_name":null,
Oct 18 21:17:08 raspberrypi workload-github.com-open-horizon-examples-wiki-service-gps_2.0.5_ae9f6194-7f1f-42cd-9137-a5b5449f7d25_gps[396]: "region_code":null,
Oct 18 21:17:08 raspberrypi workload-github.com-open-horizon-examples-wiki-service-gps_2.0.5_ae9f6194-7f1f-42cd-9137-a5b5449f7d25_gps[396]: "region_name":null,
Oct 18 21:17:08 raspberrypi workload-github.com-open-horizon-examples-wiki-service-gps_2.0.5_ae9f6194-7f1f-42cd-9137-a5b5449f7d25_gps[396]: "city":null,
Oct 18 21:17:08 raspberrypi workload-github.com-open-horizon-examples-wiki-service-gps_2.0.5_ae9f6194-7f1f-42cd-9137-a5b5449f7d25_gps[396]: "zip":null,
Oct 18 21:17:08 raspberrypi workload-github.com-open-horizon-examples-wiki-service-gps_2.0.5_ae9f6194-7f1f-42cd-9137-a5b5449f7d25_gps[396]: "latitude":null,
Oct 18 21:17:08 raspberrypi workload-github.com-open-horizon-examples-wiki-service-gps_2.0.5_ae9f6194-7f1f-42cd-9137-a5b5449f7d25_gps[396]: "longitude":null,
Oct 18 21:17:08 raspberrypi workload-github.com-open-horizon-examples-wiki-service-gps_2.0.5_ae9f6194-7f1f-42cd-9137-a5b5449f7d25_gps[396]: "location":{
Oct 18 21:17:08 raspberrypi workload-github.com-open-horizon-examples-wiki-service-gps_2.0.5_ae9f6194-7f1f-42cd-9137-a5b5449f7d25_gps[396]: "geoname_id":null,
Oct 18 21:17:08 raspberrypi workload-github.com-open-horizon-examples-wiki-service-gps_2.0.5_ae9f6194-7f1f-42cd-9137-a5b5449f7d25_gps[396]: "capital":null,
Oct 18 21:17:08 raspberrypi workload-github.com-open-horizon-examples-wiki-service-gps_2.0.5_ae9f6194-7f1f-42cd-9137-a5b5449f7d25_gps[396]: "languages":null,
Oct 18 21:17:08 raspberrypi workload-github.com-open-horizon-examples-wiki-service-gps_2.0.5_ae9f6194-7f1f-42cd-9137-a5b5449f7d25_gps[396]: "country_flag":null,
Oct 18 21:17:08 raspberrypi workload-github.com-open-horizon-examples-wiki-service-gps_2.0.5_ae9f6194-7f1f-42cd-9137-a5b5449f7d25_gps[396]: "country_flag_emoji":null,
Oct 18 21:17:08 raspberrypi workload-github.com-open-horizon-examples-wiki-service-gps_2.0.5_ae9f6194-7f1f-42cd-9137-a5b5449f7d25_gps[396]: "country_flag_emoji_unicode":null,
Oct 18 21:17:08 raspberrypi workload-github.com-open-horizon-examples-wiki-service-gps_2.0.5_ae9f6194-7f1f-42cd-9137-a5b5449f7d25_gps[396]: "calling_code":null,
Oct 18 21:17:08 raspberrypi workload-github.com-open-horizon-examples-wiki-service-gps_2.0.5_ae9f6194-7f1f-42cd-9137-a5b5449f7d25_gps[396]: "is_eu":null
Oct 18 21:17:08 raspberrypi workload-github.com-open-horizon-examples-wiki-service-gps_2.0.5_ae9f6194-7f1f-42cd-9137-a5b5449f7d25_gps[396]: }
Oct 18 21:17:08 raspberrypi workload-github.com-open-horizon-examples-wiki-service-gps_2.0.5_ae9f6194-7f1f-42cd-9137-a5b5449f7d25_gps[396]: }
Oct 18 21:17:08 raspberrypi workload-github.com-open-horizon-examples-wiki-service-gps_2.0.5_ae9f6194-7f1f-42cd-9137-a5b5449f7d25_gps[396]: 2018/10/18 21:17:08 Blue Horizon gps service configuration:
Oct 18 21:17:08 raspberrypi workload-github.com-open-horizon-examples-wiki-service-gps_2.0.5_ae9f6194-7f1f-42cd-9137-a5b5449f7d25_gps[396]: 2018/10/18 21:17:08 HZN_GPS_PORT=31779
Oct 18 21:17:08 raspberrypi workload-github.com-open-horizon-examples-wiki-service-gps_2.0.5_ae9f6194-7f1f-42cd-9137-a5b5449f7d25_gps[396]: 2018/10/18 21:17:08 HZN_USE_GPS=false
Oct 18 21:17:08 raspberrypi workload-github.com-open-horizon-examples-wiki-service-gps_2.0.5_ae9f6194-7f1f-42cd-9137-a5b5449f7d25_gps[396]: 2018/10/18 21:17:08 HZN_LAT=0
Oct 18 21:17:08 raspberrypi workload-github.com-open-horizon-examples-wiki-service-gps_2.0.5_ae9f6194-7f1f-42cd-9137-a5b5449f7d25_gps[396]: 2018/10/18 21:17:08 HZN_LON=0
Oct 18 21:17:08 raspberrypi workload-github.com-open-horizon-examples-wiki-service-gps_2.0.5_ae9f6194-7f1f-42cd-9137-a5b5449f7d25_gps[396]: 2018/10/18 21:17:08 HZN_LOCATION_ACCURACY_KM=0
Oct 18 21:17:08 raspberrypi workload-github.com-open-horizon-examples-wiki-service-gps_2.0.5_ae9f6194-7f1f-42cd-9137-a5b5449f7d25_gps[396]: 2018/10/18 21:17:08 (location source): Estimated
Oct 18 21:17:08 raspberrypi workload-github.com-open-horizon-examples-wiki-service-gps_2.0.5_ae9f6194-7f1f-42cd-9137-a5b5449f7d25_gps[396]: 2018/10/18 21:17:08 (/dev patterns): /dev/ttyACM*, /dev/ttyAMA*
Oct 18 21:17:08 raspberrypi workload-github.com-open-horizon-examples-wiki-service-gps_2.0.5_ae9f6194-7f1f-42cd-9137-a5b5449f7d25_gps[396]: 2018/10/18 21:17:08 CACHED: Source: "Estimated" -- no obfuscation
Oct 18 21:17:08 raspberrypi workload-github.com-open-horizon-examples-wiki-service-gps_2.0.5_ae9f6194-7f1f-42cd-9137-a5b5449f7d25_gps[396]: 2018/10/18 21:17:08 CACHED: {"location":{"latitude":0,"longitude":0,"elevation":0,"accuracy_km":0,"loc_source":"Estimated","loc_last_update":0},"satellites":null}
Oct 18 21:17:08 raspberrypi workload-github.com-open-horizon-examples-wiki-service-gps_2.0.5_ae9f6194-7f1f-42cd-9137-a5b5449f7d25_gps[396]: 2018/10/18 21:17:08 REST server is launching on port 31779.

SDR UI: Cloud Header does not move with page when scrolling

When the page extends past the browser's maximum height and requires the user to scroll, the top header does not move with the page.

Build issues for the example CPU code.

2 Issues discovered by Chris working through the dev guide on arm64:

• The references in the Makefile and the Dockerfiles to ../../../tools/kafkacat/$(ARCH)/ don't resolve when following Dave's recommended development procedure (copying files from open-horizon/examples over to ~/hzn. I think these paths should be: ~/examples/tools/kafkacat/$(ARCH)/. So glen will instruct the user to also copy the tools/kafkacat/$(ARCH) dir.
• In the Makefile, in build, some bash is used. Without calling bash explicitly this fails on arm64/ubuntu16.04 (TX2 current with Jetpack3.3). See fix in comment below.

Links are broken on https://github.com/open-horizon/examples/wiki

• Updated links to point to Edge Computing Documentation in the IBM Knowledge Center

Clean up speedtest2wiotp POC to work with WIoTP

• Create, test, push and publish 2 services (speedtest and speedtest2wiotp) and associated deployment pattern (speedtest2wiotp) for architectures arm, arm64, and amd64`.
• create README.md files in each service detailing briefly how to use
• Register nodes from those three architectures and verify data flow to IBM Cloud WIoTP
• Submit PR for the above code: #144
• Bruce Potter to review PR, please
• Add to the README.md in speedtest2wiotp explaining what to set up in wiotp org, instance, device type, device ID/token, API key name and token (required to run and test the code, or to register an Edge Node on this pattern)
• Create a card in the WIoTP web UI to observe this data stream, and add info into the speedtest2wiotp README.md file about how to do this.

Part of epic https://github.ibm.com/Edge-Fabric/project-summary/issues/35

Change gps service to get location from dongle by default

• Change gps service to get location from dongle by default if the device is present
• Move the global LocationAttribute config variables to userInput variables in the service definition