gns-science / solvis-api

a serverless web API for analysis of opensha modular Inversion Solutions

solvis-api

a serverless web API for analysis of opensha modular Inversion Solutions

The API openAPI/Swagger documentation is served by default from the service root.

Getting started

virtualenv solvis-api
npm install --save serverless
npm install --save serverless-dynamodb-local
npm install --save serverless-python-requirements
npm install --save serverless-wsgi
sls dynamodb install

WSGI

sls wsgi serve

Run full stack locally

npx serverless dynamodb start --stage dev &\
SLS_OFFLINE=1 npx serverless offline-sns start &\
SLS_OFFLINE=1 npx serverless wsgi serve

Unit tests

TESTING=1 nosetests -v --nologcapture

TESTING overrides SLS_OFFLINE to keep moto mockling happy

solvis-api's Issues

Feature: add Cloudwatch instrumentation

FIX: dataframe size exceeds dynamodb max (400kb)

We are creating seriaslised dataframes that too large to store as a single dynamodb object. Two potential fixes:

• store the datframe in a more space efficient means (e.g. https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.to_pickle.html )
• subdivide the dataframe into smaller chunks

Traceback (most recent call last):
  File "/Users/benchamberlain/gns/solvis-api/lib/python3.9/site-packages/pynamodb/connection/base.py", line 1023, in put_item

    return self.dispatch(PUT_ITEM, operation_kwargs, settings)
  File "/Users/benchamberlain/gns/solvis-api/lib/python3.9/site-packages/pynamodb/connection/base.py", line 329, in dispatch

    data = self._make_api_call(operation_name, operation_kwargs, settings)
  File "/Users/benchamberlain/gns/solvis-api/lib/python3.9/site-packages/pynamodb/connection/base.py", line 440, in _make_api_call

    raise VerboseClientError(botocore_expected_format, operation_name, verbose_properties)
pynamodb.exceptions.VerboseClientError: An error occurred (ValidationException) on request (f27ab3de-05d8-4ccb-bf89-d1f36d440091) on table (solution_locations_radii_dataframes) when calling the PutItem operation: Item size has exceeded the maximum allowed size

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/benchamberlain/gns/solvis-api/node_modules/serverless/lib/plugins/aws/invokeLocal/runtimeWrappers/invoke.py", line 94, in <module>

    result = handler(input['event'], context)
  File "/Users/benchamberlain/gns/solvis-api/./api/analysis.py", line 89, in handler

    process_event(evt)
  File "/Users/benchamberlain/gns/solvis-api/./api/analysis.py", line 82, in process_event
    dataframe.save()
  File "/Users/benchamberlain/gns/solvis-api/lib/python3.9/site-packages/pynamodb/models.py", line 447, in save

    data = self._get_connection().put_item(*args, **kwargs)
  File "/Users/benchamberlain/gns/solvis-api/lib/python3.9/site-packages/pynamodb/connection/table.py", line 150, in put_item

    return self.connection.put_item(
  File "/Users/benchamberlain/gns/solvis-api/lib/python3.9/site-packages/pynamodb/connection/base.py", line 1025, in put_item
    raise PutError("Failed to put item: {}".format(e), e)
pynamodb.exceptions.PutError: Failed to put item: An error occurred (ValidationException) on request (f27ab3de-05d8-4ccb-bf89-d1f36d440091) on table (solution_locations_radii_dataframes) when calling the PutItem operation: Item size has exceeded the maximum allowed size

Feature: POST call with valid ID performs multi-site pre-processing on an Inversion Solution.

We want to do the time-consuming pre-processing in advance in a serverless functions.

• time limit for serverless function is 15m
• time limit for HTTP gateway ->serverless is 30s. This is too short so we need to queue these operations

Done when:

• list of locations is available with ID in the API /location_lists
• list of radii is available in the API) /radius_lists
• Preprocessing:
  • Pass a valid Toshi Inversion Solution ID
  • fetch the solution
  • pre-process the solution, producing a Pandas Dataframe in JSON format
  • store the dataframe in DynamoDB, indexed by ID + location_list + radius_list

Bug: Auckland does not combine with other locations using "and"

When requesting ruptures though multiple locations Auckland combined with another location includes ruptures that do not intersect the Auckland radius. This does not appear to be the case for other combinations e.g. Wellington and Queenstown.

Feature: Setup CI/CD using github actions

set up for

• AWS_TEST
• AWS_PROD

BUG: Solution Ruptures not found

getting "No ruptures satisfy the query."

from PROD SW52ZXJzaW9uU29sdXRpb246MjI2NDguMEdKN3ZO

For WLG 100KM which most certainly has ruptures.

Feature: Add Kaikoura to main NZ locatoins list

see https://en.wikipedia.org/wiki/Kaik%C5%8Dura

done when:

• added to main list
• add option to process just some arbitrary (but valid location(s)) in analyser
• shows up in tests

BUG: ROT 10 causing 500 error

This query to API raises 500

https://fcx7tkv322.execute-api.ap-southeast-2.amazonaws.com/test/solution_analysis/SW52ZXJzaW9uU29sdXRpb246Mjc0OS4wMlhVTlA=/loc/ROT/rad/10?max_mag=10&min_mag=5&max_rate=1e0&min_rate=1e-20

configure serverless framework and do an initial deployment to TEST env

Feature: fetch geojson event features for a given Inversion Solution ID

Given a valid TOSHI ID

Done when

• user passes ID, radius list id (from /radius_list) and location_list id (from /location_lists)
• user passes actual radius and city
• when the ID is not found (was it pre-processed) return 404 not found HTTP error
• when the ID, and other arguments are valid return a json response containing each earthquake event (rupture) in geojson form.

Feature: migrate to row-based dynamodb model

Dealing in dataframes is not gonna work with larger datasets, and the API is a bit too slow right now for a nice UX

Feature: POST General Task ID

We want to Perform pre-processing for all IS in a given GT, just pass the GT ID.