placeos / placeos

Integrate the physical with the digital. Then automate it.

PlaceOS

This repository defines PlaceOS platform releases. It provides new release notification and the canonical reference for the platform's release history.

Platform Versioning

A PlaceOS deployment is a set of interdependent services and associated tools. Each of these have their own version number, test and release process. Platform versions overlay this to provide a set of compatible, deployable components.

A platform version looks like:

placeos-1.2104.1
        |  |   |
        |  |   revision
        |  |
        |  minor release (year, month)
        |
        major release

Major

Releases that share a major version are consistent for core features.

Moving to a new major version may include breaking changes and the removal of deprecated functionality. This includes both machine facing (API’s, infrastructure) and human facing (core platform concepts, UX).

Minor

Minor versions introduce new features that are forwards compatible.

Numbering reflects the original release packaging date (YYMM). This will perpetually increase, including across major versions.

When moving to a new minor version deprecation warnings may appear in API responses and logging, but functionality will remain.

Revision

Revisions increment within each minor version, resetting to 0. These represent change to address a bug, feature limitation or security vulnerability in the originally packaged minor release.

An increment to a release number can represent a change to one or more underlying services.

---

When operating a deployment, all components should use a consistent version.

Channels

Release channels exist as a set of dynamic tags on service images. They provide a way to pair a PlaceOS instance with a release maturity suited to the deployment environment.

nightly

Collects the state of all services based on their primary branch and builds against upstream dependencies every 24 hours. Releases on this channel are automated and should not be considered stable. These are ephemeral and do not have a corresponding platform version.

Intended primarily for development environments.

latest

Main release channel.

Recommended for production environments.

preview

The latest preview build.

NOT recommended for production environments.

Release Artefacts

Docker

Images for all services are available from Docker Hub.

Source Code

References to all source repositories exist as submodules. If you have already cloned this repo, you can access these via git submodule update --init. These reference commits used to construct the built artefacts at each release.

CI Status

Libraries

Services

placeos's Issues

Keep a CHANGELOG

Is your feature request related to a problem? Please describe.

Users (including administrators) need a view of what has changed between versions. This can include new features, as well as bug fixes.

Describe the solution you'd like

keep a CHANGELOG

Describe alternatives you've considered

Automatically generate a list of changes from the semantic commit messages of all vendored services as a PR to the release branch, where it is reviewed, edited, then merged.

Include ui-templates with release packaging

PlaceOS user-interfaces monorepo needs to be included here for linked version history.

Zigbee / Thread Integration

Build Service

Develop an independent service for reproducible builds and caching of PlaceOS drivers.

This will reduce cold start time, and increase oncoming reliability, and ease of upgrade

InfluxDB Analytics Service

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

Describe the solution you'd like
A clear and concise description of what you want to happen.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

Upgrade PlaceOS to Crystal v1.0.0

Cisco ISE Integration

.github repo for all orgs

Configure .github repo for all Place Orgs and deploy automated labelling scheme.

Update action is (incorrectly) using me as the commit author

From reading this is looks like github.actor in scheduled tasks maps to the user who last edited the schedule. This is undesired for this user case.

Will switch this over to [email protected] for better filtering of automatic dependency bumps.

Cloud Service Marketplace Apps

Cloud Service/Provider Marketplace apps to allow 1 click deployment of PlaceOS on various cloud platforms, to be triggered by deployment manager on PlaceOS Portal.

• AWS: Kubernetes
• AWS: Fargate
• AWS: OpenShift
• Azure: Kubernete/AKS
• GCP: Kubernetes
• GCP: OpenShift

Domain Admin (API Multi-Tenancy)

Create an optional domain level ownership of data
i.e. a system or zone belong to this domain - providing a path for creating domain admins / support users that have total administration access over those elements.

might be good to work on this in tandem with #33

Azure MarketPlace App

Options to deploy PlaceOS on Azure

• single server POC / development
• HA K8s cluster

Google MarketPlace App

Deployment options for PlaceOS on Google cloud

• single server POC / development
• HA K8s deployment

Native Mobile Applications

Generic, Native Mobile Apps that can be deployed directly to the Android and iOS App Stores.

These apps would be a generic, PlaceOS branded application that would expose features enabled in a customers deployment.

The customer would 'connect' the app to their deployment.

This would allow maintenance of a single app code base while delivering native features overcoming some limitations with web apps on various devices.

Internal Analytics Platform

Internal Analytics Platform to be used by Place Technology for various tasks/information.

Add caching to build action

Use action/cache to reduce build times.

Backoffice Uplift

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

Describe the solution you'd like
A clear and concise description of what you want to happen.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

Standardised Room Logic

Construction of a set of composable logic drivers that provide abstractions for common room interactions.

Forthcoming dependency action

Create a github action to check compatibility of a service against updated shards and impending versions of the crystal compiler.

ELK Logging

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

Describe the solution you'd like
A clear and concise description of what you want to happen.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

Implement more granular OAuth scopes

Is your feature request related to a problem? Please describe.
In order to provide tighter access control for applications built on PlaceOS, API interactions need to restrict behaviour based on the current auth scope.

This works alongside the current user based access control, rather than replacing it, to enable users to request an auth token that provides a limited set of functionality based on usage needs. For example, when an admin user is interacting with a staff app or visitor UI, the access token used should not have full privileges for system management.

Describe the solution you'd like
Clearly defined and managed auth scopes for all external API endpoints.

Describe alternatives you've considered
NA

Additional context
https://oauth.net/2/scope/
https://www.oauth.com/oauth2-servers/scope/defining-scopes/

UI Web Content Accessibility

https://www.w3.org/WAI/WCAG21/quickref/

Platform Release Process

Release definitions are established, need to develop correct process including documentation and change logs.

This feature to be completed once Release Testing Feature is complete.

Issue on page Repositories

Describe the bug
"Pull the latest commit for this repository" on Place OS Drivers repository fails and does not show a clear error or reason for the failure.

To Reproduce
Steps to reproduce the behaviour:

1. Go to Repos
2. Click on Place OS Drivers
3. Click on Pull
4. After a little while see error: "Error pulling latest commit. Error: {}"

Expected behaviour

1. The repo is pulled successfully without error; or
2. If there is an error in pulling the repo, display information about the error which can help the user identify and rectify the problem.

Screenshots

Desktop (please complete the following information):

• OS: Windows 10
• Browser Chrome Version 86.0.4240.183 (Official Build) (64-bit)

Additional context

Hash: b746e20
Built: 09 Nov 2020, 8:42 AM

AWS MarketPlace App

Options to deploy PlaceOS on AWS

• Single sever POC
• K8s Cluster
• Fargate? - we need to look into a nice way of migrating a clients instance to the latest version, which might be hard to automate using fargate

Image scans

Could we extend the github actions to include a vulnerability scan of the resulting docker images?

Can use something like this: https://github.com/Azure/container-scan

Basic Room Settings UI

Describe the solution you'd like
Apply basic room settings and configuration via a UI rather than entering system configuration code.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

Develop Standardised Front End Testing Pack for Partners

Develop a standardises automated front end test pack against templated user interfaces using Katalon Studio, made available for partners as a building block towards testing their own PlaceOS interfaces where the standard template has been used.

Support build retries

There are a number of failure modes than can impact service image builds which may be due to external factors as opposed to something incorrect with the build / repo responsible for this. Examples are: an external resource that's unavailable or intermittent compiler errors.

To provide accurate CI status, and remove the need for manual intervention, builds should retry a set number of times if this is encountered.

Contact Tracing

Code for querying InfluxDB to obtain location data and map datapoints that have been in proximity
needs to cross-reference the data in the Location Services modules to identify people (influx doesn't store this)
frontends need to be built for displaying the data

The Magic EventBus

We currently use RethinkDB as the database as it has a built-in Event Bus feature.
Allowing services to listen for database changes so those changes can be applied to the service

The plan is to migrate from RethinkDB to Postgresql which would allow us to leverage cloud hosted database services, however we need to replace the Event Bus feature available in RethinkDB with something that works with vanilla Postgresql

A service that runs in each region for providing a local source of database change events and a crystal client that replicates what is provided by the RethinkDB client. To minimise the code changes required to adopt the new service.

Render the CI status badges for services

Render the CI status badge for each service submodule in the README.
This could expand into a simple table with metadata for each service, like the latest tag, build status, the commit that it's on etc.

Debug Console Not Printing Debug

Describe the bug
When selecting a driver to debug in a system, the debug modal opens but no debug data is printed. It is also observed that no debug data is printed to the browser console.

Your environment
This issue appears to have started post update to 1.2104, version specific details:

To Reproduce
Select a driver to debug. Either monitor or exec a function on the driver and observe no debug data is printed.

Expected behavior
Device debug and general device comms should be printed in debug view.

Screenshots

E2E functionality tests

#3 introduces a test suite that can be run as a requirement to merge into nightly.

Perhaps we should have an unstable branch that must first pass the integration pipeline before a merge into the nightly branch.

Front End Builder App

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

Describe the solution you'd like
A clear and concise description of what you want to happen.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

Cisco Webex Native Integration

Provide native integration to the Webex Teams App to allow users to identify and book a suitable room for a future or adhoc meeting. Potentially through the use of a PlaceOS Bot.

• People Count (amount of people in the room)
• Visitors attended vs. Joined virtually
• Visitor arrival notifications to Webex Teams
• Webex teams status in Workplace App

Use crystal-client submodule

release is supposed to imprint the state of all placeos libraries every time an update is produced. It would be more appropriate then for shards to reference crystal-client submodule inside release itself for specs and other purposes instead of the remote crystal-client on github (which is the current configuration at the moment in shards.yml).

Apple/Google Wallet Integration

Allow visitor passes and QR Codes for entry access systems to be added to Apple/Google Device wallets after the user has checked in using the Enrolment App.

Driver Migration

Migration of Ruby Engine Drivers to PlaceOS

Hot list dashboard (offline devices etc)

Is your feature request related to a problem? Please describe.

We need something to replace the Metrics dashboard that was available in Engine.
The primary user story would be "I want to see the list of offline devices in this building"

Describe the solution you'd like

We should leverage the MQTT streaming and the NGINX JWT authentication plugin
so the frontend can connect to MQTT via websockets protected by JWT auth.

Then using the MQTT filters: https://docs.google.com/document/d/1gBZD296sF0cZXYyRKrRjn-wi5c408Yg1189S-DV9JgE/edit

Allow people to filter by zones, however also filter the state they are interested in.
i.e. connected state could be the default (which devices have issues), but maybe you are interested in the meeting rooms in use right now on a particular floor or which security doors are open or lights are on etc

This requires a Mosquito MQTT websocket endpoint with the NGINX JWT token filter for authenticated access.

Builds failing due to cache issues

Nightly builds have recently started failing due to a failure in the cache cleanup step. This will likely impact release builds too.

Example failed run: https://github.com/PlaceOS/PlaceOS/actions/runs/720052976.

Deployment via OpenShift

Support OpenShift as a viable deployment and hosting method for the PlaceOS Platform including development of necessary helm charts to make existing Kubernetes deployments compatible with OpenShift.

Self Documenting API

As PlaceOS grows and changes, our APIs do too.
CI tooling to extract up to date documentation for each API external or internal becomes important in ensuring visibility of product surface for members of other product teams, frontend teams, and our partners.

Targets

• plain ol' markdown - ala crystal docs
• openAPI specification
• Apiary (or other fancy API documentation renderer) via OpenAPI specification

Extraction

openapi-generator looks quite mature, implementation of an action-controller RouteProvider would be a great first step in documenting API surface

The above can extract information from accesses to the query context, as well as calls to render.
As a part standardising interface development, supporting definition of a params context via http-params-serializable would be excellent. Worth investigating interop

Failed builds can result in inconsistent state on docker hub

Builds should only push to docker hub after all images have successfully built and tests have passed where appropriate.

Currently a failed service image build will cancel others, but as these actions happen in parallel a subset of images may have already been published.

Jobs for building a publishing need to be separated and sequenced.

Include `edge` service in release packaging

Add service definition for building edge (https://github.com/PlaceOS/core/tree/feat/edge).

Release Testing Suite

Integrated/automated tests on release version.

Overarching Plan: https://docs.google.com/document/d/1FFUcNcLReh9y9PtQXkzrsgUPQiNUvdJQnfZqjkNEKRI/edit#

Analytics Connectors

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

Describe the solution you'd like
A clear and concise description of what you want to happen.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

PlaceOS Log Management

Tracking issue for improving the logging and visibility of PlaceOS

Verify inter-service dependencies before publishing release

Encapsulate release tasks in a docker container that will be run as a verification action before cutting a release.

Useful tasks

• Verify all services have a specific shard within a semver range
• Verify all services built against the same crystal compiler

MS Teams Native Integration

Provide native integration to the MS Teams App to allow users to identify and book a suitable room for a future or adhoc meeting. Potentially through the use of a PlaceOS Bot.