gns-science / nshm-toshi-client Goto Github PK

As a ToshiAPI user, I want to access any file-like object via in my python environment using NTC so that it's easy to retrieve file metadata and the file object itself.

Done when:

• Use NTS to retrieve file attributes from any ToshiAPI objects supporting FileInteface (eg. File, InversionSolutoin, ScaledInversionSolution, OpenquakeHAzardOutput)
• optionally get meta k,v pairs
• optionally get file_url
• test coverage
• include example in 'docs/usage.md'

Notes

The module toshi_api runzi.automation.scaling.toshi_api contains some useful file queries that should be ported to nshm-toshi-client (NTC)...

get_file_detail(self, id)
get_file_download_url(self, id):

These are very simple and useful all their features should be in the NTC. And Runzi should be ported to use these from NTC once they're ported.

We need a a new release with updated core libraries

For general tasks that are subtask type: SCALE_SOLUTION the following query results in an error:

can be demonstrated using ID R2VuZXJhbFRhc2s6MTA1NTc0 in PROD

qry = '''
            query one_general ($id:ID!)  {
              node(id: $id) {
                __typename
                ... on GeneralTask {
                  id
                  title
                  description
                  created
                  swept_arguments
                  children {
                    #total_count
                    edges {
                      node {
                        child {
                          __typename
                          ... on Node {
                            id
                          }
                          ... on AutomationTaskInterface {
                            created
                            state
                            result
                            arguments {k v}
                          }
                        }
                      }
                    }
                  }
                }
              }
            }'''

        # print(qry)
        input_variables = dict(id=id)
        executed = self.run_query(qry, input_variables)

File ~/.virtualenvs/runzi/lib/python3.8/site-packages/nshm_toshi_client/toshi_client_base.py:69, in ToshiClientBase.run_query(self, query, variable_values)
     66 if self._with_schema_validation:
     67     self._client.validate(gql_query) #might throw graphql.error.base.GraphQLError
---> 69 response = self._client.execute(gql_query, variable_values)
     71 #logger.debug('response: %s', response)
     73 if response.get('errors') is None:

File ~/.virtualenvs/runzi/lib/python3.8/site-packages/gql/client.py:78, in Client.execute(self, document, *args, **kwargs)
     76 result = self._get_result(document, *args, **kwargs)
     77 if result.errors:
---> 78     raise Exception(str(result.errors[0]))
     80 return result.data

As a ToshiAPI user I want the nshm-toshi-client NTC to automate using the standard API ENV variables so that less boilerplate code is required...

Done when:

• API changes are agreed
• test coverage added/updated
• ENV variables are covered in docs/useage

current:

>>> from nshm_toshi_client.toshi_file import ToshiFile
>>> import os
>>> API_URL  = os.getenv('NZSHM22_TOSHI_API_URL', "http://127.0.0.1:5000/graphql")
>>> S3_URL = os.getenv('NZSHM22_TOSHI_S3_URL',"http://localhost:4569")
>>> API_KEY = os.getenv('NZSHM22_TOSHI_API_KEY', "")
>>> headers={"x-api-key":API_KEY}
>>> api = ToshiFile(API_URL, None, None, with_schema_validation=True, headers=headers)

proposed:

>>> from nshm_toshi_client import ToshiFile, API_URL, API_KEY, S3_URL
>>> headers={"x-api-key":API_KEY}
>>> api = ToshiFile(API_URL, None, None, headers=headers)

or

from nshm_toshi_client.toshi_file import ToshiFile
from nshm_toshi_client.config import API_URL, API_KEY, S3_URL

As SD team

We want to demonstrate the API / client features using StrongMotionStation

So that we get some end user feedback from GMCM team (CVH, LW, AK) re its utility

Done when:

• filter SMS data into a dataframe with a Jupyter Notebook
• demo script to upload all stations from the main spreadsheet
• demo script to upload files and link to SMS node (by station code) from a local folder

with incorrect bucket URI (e.g. TOSHI_S3_URL=https://nzshm22-toshi-NOT-api-test.s3.amazonaws.com

the script line

self._ruptgen_api.upload_task_file(task_id, outputfile, 'WRITE', meta=task_arguments)

doesn't raise an error

• get the all the files , givn a parent task ID
• get a download link given a file id

As a user I want the client to provide me a simple download helper so that I can fetch files as easily as possible from the ToshiAPI, and it

• should also let me rename the file
• should let me specify a path to save to

NB both these can be supported with similar logic to posix mv command .i.e

When you specify a single source file and the target is not a directory, mv moves the source to the new name, by a simple rename if possible.

Done when:

• the client has a file_download helper function
• uses the usual ENV variables by default
• supports saving to specific output folder
• supports renaming the file
• If user specifies a ToshiID that does not support FileInterface then throw a helpful exception.
• on success return the final name of the saved file.

• version: 0.6.0
• Python version: 3.9
• Operating System: Linux

Description

As per @chrisdicaprio description

What I Did

updated to latest nshm-toshi-client and used the api. When ToshiAPI client passed with_schema_validation=True (as almost all client do), the gql.client error throws assertion. error