resonantgeodata / rd-oasis

ResonantGeoData OASIS Deployment

RD-OASIS

What is OASIS?

OASIS is a Django application designed to allow users to define and run arbitrary algorithms, tracking the progress and results of such algorithms in Django.

The Algorithm Execution Lifecycle

These are the steps that are taken by tasks to execute an algorithm. This is generalized between Kubernetes and Celery, as the steps are the same at a high level.

Main Components

These are the main components that make up OASIS. Each of these components is a Django model, and each entry here will contain a short description, followed by the fields associated with each.

Algorithm

Algorithms are at the core of OASIS, and contains much of the required definitions for any process you want to run.

• Name - The name of the Algorithm
• Docker Image - The docker image (documented below) that this algorithm will use.
• Command - The command used to invoke your algorithm
• Entrypoint - If necessary, override the default entrypoint of your docker image.
• Environment - Any environment variables that should be passed into the container when running your algorithm.
• GPU - Whether GPU access is required by this algorithm.

Algorithm Task

Algorithm tasks are individual runs of an algorithm. Algorithm tasks are isolated from each other, only sharing the underlying algorithm itself.

• Algorithm - The algorithm which this task belongs to.
• Status - The current status of this task (its progress in the task lifecycle). One of:
  • Created
  • Queued
  • Running
  • Failed
  • Succeeded
• Output Log - The output (stdout) of the algorithm, stored as a text field.
• Input Dataset - The dataset containing the input, to be mounted to and copied into the /<working_dir>/input directory.
• Output Dataset - The dataset containing any files produced by the algorithm (any files placed into the /<working_dir>/output directory).

Docker Image

This is the defintion of the image to be used when running your algorithm. This docker/container image contains the necessary environment to run your algorithm. Generally, any necessary files, libraries, packages, etc. that are required by your algorithm to run, are included in this image.

• Name - The name of the image
• Image ID - The id of this image on Docker Hub.
• Image File - If this image is uploaded directly to the API, instead of on docker hub, then this field points to the file which contains it.

Dataset

A Dataset is a container of files to be used by algorithms. This is used to facilitate both input to and output from an algorithm task.

• Name - The name of the dataset.
• Files - The files contained within this dataset.
• Size - The size (in bytes) of this dataset.

rd-oasis's Issues

Generally refactor/cleanup the algorithms app

Rethink these algorithms models and clean them up - the goal is to remove as much technical debt as possible

We should also pull in @AlmightyYakob to think about Danesfield project needs here since I believe there are overlapping goals (perhaps this algorithms app will be made into a re-usable app)

Export any vtkPolyData (point clouds) to 3D Tiles

@danlipsa

cc @aashish24

Meta issue to track the task of updating the VTK writer for 3D tiles to handle any vtkPolyData

Add GPU allocation specification in kubernetes job definition

Currently, the GPU isn't requested for jobs. This should be added, so containers that need GPU can run.

https://kubernetes.io/docs/tasks/manage-gpus/scheduling-gpus/

Prototype XAI integration into OASIS to showcase saliency maps

Assess deployment options

Should we deploy this on Heroku (once in a stable state and we have a demo)?

Add informative landing page on the OASIS app

Deploy OASIS back-end infrastructure on AWS (worker nodes)

Blog Posts!

Enable GPU execution of Algorithms

Enable users to run the model on GPU or CPU

This will likely be implemented as a field on the Algorithm model such as use_gpu with the options no, optional, and required

Gather new snow (ASO) data from LBNL

Reminder to collect additional aerial snow data from LBNL and corresponding ML model.

Use RGD's ChecksumFile (or Raster) for data fields

The tasks here use S3FileField directly. We should use RGD's file models... or if we know tasks will be running on rasters, they should point to Raster so that the input/output will be searchable in RGD's UI

Improve README and usage instructions

Create documentation on how the OASIS backend system works (including a diagram) and how to setup it up (devops).

CI and Testing

Implement tests for the algorithms app and run those tests on CI

Fix running docker in docker

Add tests

There's many tests needed for the algorithms app, which have yet to be implemented.

Add specific dataset file add/remove endpoints

Currently, the only way to modify which files are in a dataset is to submit a PUT, PATCH, or DELETE request to the dataset detail view, which specifying all of the files which should now belong in the files field. For some reason, this is very slow. An approach that might be more performant is to add an endpoint for adding/removing files. I was thinking of the following structure for the request payload:

{
  "insert": [
    1,
    2
  ],
  "delete": [
    12,
    21
  ]
}

Additionally, a /clear endpoint could be added onto the dataset view, which as a convenience, would just clear the dataset in one request, with no payload.

Develop PyTorch data loader for RGD and demonstrate its use

Ref https://github.com/ElementAI/seasonal-contrast

more details to come

Demo: image processing Algorithm

Generate a demo of this workflow to process a raster/image and produce a processed image/raster output

LBNL Use Case

Utilize changes in ResonantGeoData/ResonantGeoData#457 for the Jupyter notebooks that do image resampling and mosaicing

Deploy Vue app on Netlify

Deploy OASIS API/DB using Heroku

Add tree endpoint for Dataset ViewSet

This should function the same as the tree endpoint for checksum files, but only applied to files in a particular dataset. The endpoint would be /datasets/{id}/tree.

Ingest output files into corresponding RGD models

This is required for proper visualization.

Related work: https://github.com/girder/Danesfield/blob/f5d798f695465572b08fc4cdd79234152277be35/danesfield/core/tasks/__init__.py#L28-L64

Create terraform module for easy integration and re-use

Setup auto-scaler for OASIS worker nodes

Handle custom containers

Currently, custom containers (one's that are uploaded to django directly) aren't handled, only containers stored on dockerhub are. IMO the way to handle this is to use Amazon ECR to store these custom containers. Then the Kubernetes Job would specify that the image should be pulled from this repo, instead of dockerhub. Here are some kubernetes docs regarding this.