linuxforhealth / healthos

LinuxForHealth Core Services

LinuxForHealth HealthOS

LinuxForHealth HealthOS provides an interoperable open-sourced health stack for developing highly available, standards compliant, and secure applications and services.

The HealthOS project is a mono-repo with source code split into separate modules:

• core: Base level module used for data acquisition, data processing, and data distribution.
• Additional modules are coming soon!

Development Dependencies

The HealthOS development environment depends on the following 3rd party packages:

• Python 3.10 or higher the HealthOS language runtime.
• Poetry for packaging, dependency management, and publishing support.
• GNU Make for module builds.
• Pyclean removes Python bytecode, used as a cross-platform tool in the build process.

Quick Start

The LinuxForHealth HealthOS project is composed of multiple modules, where each module is located within a separate HealthOS subdirectory. Each module utilizes the same build and development tooling to improve ease of use and simplify automation.

First use make to build the development environments

make dev-env

Once the build is complete, it's time to review the HealthOS Documentation.

Documentation

• Development Tools
• Module Docs
  • Core Module
• Build Documentation

healthos's Issues

Create Vagrant VM to Test Build

Create Core Service Startup Including Fast API App

Integrate NATS into HealthOS Installation

Integrate NATS (binary release) into the HealthOS installation as a core component.

Implement Kafka Inbound Connector

Create Docker Image To Support Build Process

Complete Manual Component Installation on Vagrant VM

Core Module Enhancements

The core service currently performs the following tasks:

• classifies incoming data, raises an error if data cannot be classified.
• validates incoming data, raises an error if data is invalid.
• creates a simple "envelope", or publish data model, for processing results, prior to publishing to NATS.

In this epic we will update the core module to:

• be lenient in regards to classification and validation by allowing configurations to prefer warnings rather than errors
• include additional metadata fields for errors, warnings, provenance, etc
• have well defined "status" values in the model

Document Publish Data Model

Document the Publish Data Model used with the HealthOS core service for publishing. Once complete the documentation should reside within the core module's markdown "docs". This item is a pre-requisite before working on other issues within the Epic.

Test HealthOS Installation on s390x

Test the HealthOS installation on the s390x architecture to validate HealthOS services.
Complete this task once #40 is complete.

Integrate logging support

Update HealthOS Installation to Use systemd

Update the project and installation process to support running each HealthOS service using systemd

Create and Implement LFH Core Repo

Ubuntu 22.02 Deployment

Support deploying HealthOS modules onto Ubuntu 22 using the default package management system, apt-get. This epic supports the creation of local apt-get packages which can be manually installed onto the OS.

At a high level the package build process should "logically" include:

• The python wheel file generated from the HealthOS Makefile
• The "frozen" third party requirements generated from the module's poetry lock file
• "scripting" to create the virtualenv, install frozen requirements, and the wheel
• integration with systemctl to run the module as a service
• requirement to install Python 3.10 or higher

Finally, installing the components manually on a vagrant VM may be a beneficial exercise as it allows us to validate our "desired" state.

Add Error Subject to Internal NATS System

File Watcher Connector

In this task we will add a File Watcher Connector to the HealthOS inbound connectors to support a file based ingest. At a high level this connector will "watch" for files given a supported configuration, read files once they are available, and then transmit them to the NATS "ingress" subject provided by the Core NATS Jestream client.

Create service stop endpoint

Create Nats Setup Script

Support ingress validation

this was included as part of #25 . Validation is provided via the "detect" module which allows it to be plugged into connectors.

Create docker compose services

Integrate HDP with LFH Core

Create /ingress connector endpoint

Create service start endpoint

Update Makefile to Support Package Target

Create a "package" make target which will create a tarball containing each module's wheel, requirements.txt and configuration file. We will also need an "install" script to support the creation or users, groups, virtual environments, etc.

Tasks:

• Include module wheel files and requirements in tarball
• Create user and groups
• Install dependencies (python, systemd, etc)

Edited: moving the following tasks to separate issues

• Install NATS (ensure we are on supported on s390x) #38
• Configure systemd to run HealthOS services
• Test on s390x - #39

Create service listing/status endpoint

"Orchestration" Module

Admin Management API

The LFH HealthOS Core Admin API supports the following functions:

• listing available services or components with a status (started, stopped, error)
• starting a stopped service
• stopping a running service

The Admin API is "mounted" within the default Fast API application under the /admin endpoint.

Core Module Inbound Connectors

The LinuxForHealth HealthOS uses connectors to receive and transmit data. This epic supports the development of inbound connectors used to acquire data. Connectors in scope for this epic include:

• KafkaConsumer (uses aiokafka dependency)
• NatsClient (uses nats.py dependency)
• /ingress RestEndpoint (uses fastapi dependency)

The KafkaConsumer and NatsClient "pull" data from external sources while the /ingress endpoint allows external clients to "push" data to the HealthOS.

These connectors are configured as part of the Core Service Definition

Each of these connectors utilize asyncio based libraries to integrate with their respective data sources. As a result we will need to provide an "event loop" to the HealthOS so that these libraries can execute concurrently.

To keep things simple in this first iteration, we will create a Fast API application as a default component that is spun up during "startup". The app will support an "admin" api for interacting with connectors (implemented as asyncio tasks), and an optional "/ingress" connector endpoint. This allows us to use the event loop that Fast API provides. Due to this detail, the Fast API issues will be the first issues worked as it is provides the execution environment for the other connectors.

The flow for data ingress uses Nats subjects and subscribers for validation:

• connector receives data
• connector publishes data to lfh.healthos.ingress
• Nats client subscribed to lfh.healthos.ingress validates the data
• Data validation errors are sent to lfh.healthos.ingress.errors

Success criteria (scoped to development environment):

• The development environment includes containers to support "external services"
• The development environment includes a startup process to launch the core service using the core service configuration.
• The development environment includes tooling to transmit HL7, FHIR, X12, and dicom payloads to the inbound connectors.
• The development environment includes a logging facility to monitor the movement of data throughout the system
• Validation errors are transmitted to lfh.healthos.ingress.errors when appropriate

Implement Nats Inbound Connector

Create tooling to transmit sample messages for dev

Audit Module