nasa-pds / pds-api Goto Github PK

This requires:

• pick a use case
• ingest PDS4 archive with data files in pds-gamma registry
• make the python client work in jupyter notebook
• develop the full use case, with data access through http

As a user, I want to be able to select the fields I want in my search response of collections.

The management of null fields should be handled:

• when no fields are selected, all available fields are returned (no null fields)
• when fields are selected, all selected fields are returned. When they are not available they are set to a "null" value. This value should not be mis-interpreted for an original "null" in the pds model (to be detailed).

Acceptance criteria:
Check that specification is clear about the syntax to select fields in the JSON response, see:

1. https://github.com/NASA-PDS/pds-api/blob/master/docs/spec/pds-api-specification.md
2. and online https://nasa-pds.github.io/pds-api/
3. validate also in reference implementation https://github.com/NASA-PDS/registry-api-service deployed on https://pds-gamma.jpl.nasa.gov/api/swagger-ui.html

The api end-point should answer in json or XML.

For XML response, the original PDS4 label should be provided

Acceptance criteria:
Check that the API end-point support standard content negociation as describe here https://restfulapi.net/content-negotiation/ (especially header parameter Accept=application/json or application/xml or application/pds4+xml)

Define a PDS API extension model / convention for discipline-specific search engines

How should PDS handle enriched / supplemental metadata?

• Product_Observational_Supplemental?
• Table data that is tagged as non-archival?
• Define new search/operations table data?
• Should this be ingested into the registry? or is this separate from registry and is for search specifically? separate "section" of response format to denote supplemental or additional metadata that is non-archival?

Motivation

...so that I can ingest supplemental metadata into the registry that is both archival (Product_Metadata_Supplemental) and non-archival (e.g. external databases) to make the information searchable and accessible.

Additional Details

Note: This ticket is the parent epic for the overarching design consideration, there are several sub-tickets for the individual data source implementations.

Some use cases we need to think about:

1. Use the PDS4 product Product_Metadata_Supplemental to revise data in products in the archive. NOTE: Need to check with RMS whether or not these should be considered archival.
2. Non-archival metadata - this is "augmented" or "supplemental" metadata that is generated after archiving that can help for search and/or improves the existing archive metadata. e.g. image tags indicating a crater exists in the image, or more detailed target information like Mars - Olympus Mons, or tracking service information like "here is this products DOI that was never registered)"
   a. should we just treat these the same as Product_Metadata_Supplemental? (depends if those are considered archival or not)
3. PDS3 databases - how do we integrate with databases that contain PDS3 data?
   a. should we just treat these the same as Product_Metadata_Supplemental? (depends if those are considered archival or not)

Acceptance Criteria

See sub-tickets for individual data source acceptance criteria

Engineering Details

Design thoughts

Should we just use Product_Metadata_Supplemental as the model for this and then augment for the non-archival data sources? - it seems like we should require some sort of archive-worthy definition of the data to be ingested. this could almost be a one-to-one translation of Product_Metadata_Supplemental, but just make it non-archival.

For the external databases, and initial workaround, we may also be just using a CSV for the time being: NASA-PDS/registry#51, and implement a MySQL plugin down the road since it would require some additional data modeling and configuration to connect to the database.

💪 Motivation

...so that I can easily integrate the PDS API with other APIs.

So that PDS datasets are viewed by google as datasets, see for example

📖 Additional Details

We want to be able to add a piece of javascript code in our HTML dataset landing page (bundle, collection, observational_products), which will call the registry-api with the lidvid of the current product as an argument and complete the schema.org json-ld description of the dataset (e.g. https://schema.org/Dataset) as described on this page https://developers.google.com/search/docs/appearance/structured-data/dataset#example from the registry-api response content.

The request will use a specific Accept header value application/ld+json in the content negociation.

⚖️ Acceptance Criteria

Given
When I perform
Then I expect

⚙️ Engineering Details

See introduction https://schema.org/docs/data-and-datasets.html

Bundle/Collection labels: https://schema.org/DataCatalog
Observational product labels: https://schema.org/Dataset
Data: https://schema.org/DataDownload

As a API developer, I want to be able to test that my API complies with the specification from a ready made test collection in POSTMAN.

Acceptance criteria:
Check that specification is clear about the syntax of properties in JSON response, see:

1. https://github.com/NASA-PDS/pds-api/blob/master/docs/spec/pds-api-specification.md
2. and online https://nasa-pds.github.io/pds-api/
3. validate also in reference implementation https://pds-gamma.jpl.nasa.gov/api/swagger-ui.html

The server stub will be automatically generated from swagger file so to be updated when swagger file is.

The implementation is java.

Define initial prototype set of discipline/node-specific search parameters for API Spec

choose between null, nan or not having the attribute or other options

Acceptance criteria:
Check that specification is clear about the management of empty fields or attributes, see:

1. https://github.com/NASA-PDS/pds-api/blob/master/docs/spec/pds-api-specification.md
2. and online https://nasa-pds.github.io/pds-api/

Document SwaggerHub CodeGen Server adaptation and integration how-tos

For general documentation:
https://nasa-pds.github.io/pds-api/

Possibly for implementation documentation, for example:
https://pds-gamma.jpl.nasa.gov/api/swagger-ui.html

We want to keep the original style because user's are used to it since it comes from generic popular documentation generation but we want to show that it is a PDS documentation (at least logo) and provide links to the main web resource of PDS (app bar ?).

Core services of the API provided in the template. Basic infrastructure services will be built in

Acceptance criteria:

1. A basic implementation of the API is available open source
2. the implementation is publicly deployed online and documented

Acceptance criteria:
Check that the default fields are described in the specification:

1. https://github.com/NASA-PDS/pds-api/blob/master/docs/spec/pds-api-specification.md
2. and online https://nasa-pds.github.io/pds-api/

Design how query passing will happen across the PDS both EN-to-DN and DN-to-DN

Get the investigation area from the xml label, rewrite how target and instrument are read, get them also from the XML label

Acceptance criteria:
Check that target,instruments and investigation area are available in reference implementation https://github.com/NASA-PDS/registry-api-service and deployed on pds-gamma (https://pds-gamma.jpl.nasa.gov/api/swagger-ui.html)

• Refine format conventions defined by the API WG
• What are the default return fields?
• What are the required return fields?
• How do we split out "default" from "additional" metadata. We need to make sure the "additional" metadata fields returned do not come off as secondary citizens
• Update API Spec appropriately
• Implement in API Service

We need to come up with some intra-discipline (e.g. Imaging data) search parameters to include in the API.

Some questions:

• at what scope should we include this?
• should this just be custom query parameters (e.g. filter_name, filter_number, etc. == filter)
• for the parameters, should we define a new schema (e.g. custom/filter or search/filter ?)

Acceptance criteria:

Check that required fields are defined in the specification:

1. https://github.com/NASA-PDS/pds-api/blob/master/docs/spec/pds-api-specification.md
2. and online https://nasa-pds.github.io/pds-api/

or check it is available

Identify it and test it.

...so that we can utilize the PDS Client and test the API at "any time" against some deployed version with some data in it.

aka. CI/CD of the latest API server

Acceptance Criteria

Given I've made a change to the source code of the PDS API Service (ignoring docs)
When I perform a commit in Github
Then I expect to trigger a build and deployment to pds-gamma (and/or AWS if that is also reasonable)

Engineering Details

• Initial tweaks of ES query result weighting per NASA-PDS/registry-api-service#5
• Develop some simple docs for how we can perform some additional tweaks down the road per #93

Related Requirements

🦄 NASA-PDS/registry-api#460

Currently the non existing field is returned in objects with value=null.

We should return error 400 (bad request) or 404 (non existing).

OGC EDR chose error 400, see https://github.com/opengeospatial/ogcapi-environmental-data-retrieval/blob/master/19-086.pdf

The demo API should support the query syntax described on https://github.com/NASA-PDS/pds-api/blob/master/docs/spec/pds-api-specification.md#query-syntax in parameter 'q'.

The wildcard (*, ?) do not need to be supported in this version.

Acceptance criteria:
Test simple comparisons and combination of comparisons as documented in https://nasa-pds.github.io/pds-api/#tag/collections

You can use demo API deployment available on https://pds-gamma.jpl.nasa.gov/api/swagger-ui.html or deploy your own as descrbed in pds-registry-app package documentation version 0.3.0.

The api end-point should answer in json or XML.

For XML response, the original PDS4 label should be provided