rtisma / kf-portal-etl-task-service

:white_check_mark: A template for a microservice written in Java and a resource server under Ego with JWT authorization

Kids First ETL Task Runner

Microservice to execute ETL as a FSM process when requested by the Kids First Release Coordinator

Table of Contents

• Introduction
• Process
• Requirements
• Quick Start
• Testing
• API
• Acknowledgements

Introduction

This application will initiate ETL tasks as requested by the Kids First Release Coordinator. Tasks are modeled as Finite State Machines (FSMs) and specific state transitions (initialize, run, publish) can be started by HTTPS messages from the Release Coordinator.

ETL processes are run in docker containers which are created when the task is Run, and terminate when the ETL is staged. This allows multiple tasks to be run simultaneously in distinct containers. The ETL publish step is performed by the Java Application.

Although this application is built to accomplish the specific work of the KF ETL, the this is built with a configuration model that makes it easy to adopt any other dockerized task that the release controller wants to manage.

Authorization for requests made to the Task Runner is performed via JWTs from an EGO server. A valid token with Admin permissions for a recognized Task Coordinator application is required for any action to be taken on any request to this ETL Task Runner.

Task Model

Each Task is and FSM with the following state-transitions (yellow and green boxes), which can be initiated by the given requests (white diamonds).

The possible Task Commands from the Release coordinator are: Initialize, Status, Run, Publish. These steps match the requests outlined in the Kids First Task Coordinator documentation.

1. Initialize - This will check if the task runner is able to run a new task. Any checks that are needed ahead of starting a task are performed here. For any ETL tasks, this will include checking permissions to access the studies that will be requested in the ETL feed. On success this will return a unique task code to be used for this task.

2. Run - Start the task running and stage the results. Given ID for a task currently in PENDING status from the Intialize step, and any required variables for the task, a docker container will be created and the task run.

3. Publish - Given a task ID that is currently in STAGED status after completing the Run step, this will publish the results.

Requirements

The application can be run locally or in a docker container, the requirements for each setup are listed below.

EGO

A running instance of EGO is required to generate the Authorization tokens and to provide the verification key.

EGO can be cloned and run locally if no public server is available.

Local

• Java 8 SDK
• Maven

Docker

• Docker

Quick Start

Make sure the JWT Verification Key URL is configured, then you can run the server in a docker container or on your local machine.

Configure JWT Verification Key

Update application.yml. Set auth.jwt.publicKeyUrl to the URL to fetch the JWT verification key. The application will not start if it can't set the verification key for the JWTConverter.

The default value in the application.yml file is set to connect to EGO running locally on its default port 8081.

Run Local

$ mvn spring-boot:run

Application will run by default on port 1234

Configure the port by changing server.port in application.yml

Run Docker

First build the image:

$ docker-compose build

When ready, run it:

$ docker-compose up

Application will run by default on port 1234

Configure the port by changing services.api.ports in docker-compose.yml. Port 1234 was used by default so the value is easy to identify and change in the configuration file.

Testing

TODO: Additional instructions for testing the application.

API

The task runner provides 2 services that can be called by the Release Coordinator:

• /status - General health and version status of the Task Runner
• /tasks - Commands to create and interact with tasks:
  • Initiate new task
  • Begin Staging or Publishing existing Task
  • Query Status of Task by task ID

Api Specifications can be found here.

Acknowledgements

Services provided to accomplish the task coordination process as defined in the Kids First Task Coordinator

JWT Authentication model provided by Overture's Ego service. This microservice is built on a fork of the Ego Microservice Template for Java