nasa-pds / pds-api

PDS web APIs specifications and user's manual

NASA PDS Federated API

PDS strives to develop REST-ful web APIs for all online web applications, including data search and DOI management.

This repository is the entry point for the NASA PDS Federated API specifications (e.g. following OpenAPI standard) and their documentation.

Please visit our website at https://nasa-pds.github.io/pds-api/ for user documentation.

Prerequisites

• Python 3

• For PDF generation, basiclatex and accompanying libraries are needed

  brew install basictex
  sudo /Library/TeX/texbin/tlmgr update --self
  sudo /Library/TeX/texbin/tlmgr install latexmk
  sudo /Library/TeX/texbin/tlmgr install tex-gyre
  sudo /Library/TeX/texbin/tlmgr install fncychap
  sudo /Library/TeX/texbin/tlmgr install wrapfig
  sudo /Library/TeX/texbin/tlmgr install capt-of
  sudo /Library/TeX/texbin/tlmgr install framed
  sudo /Library/TeX/texbin/tlmgr install needspace
  sudo /Library/TeX/texbin/tlmgr install tabulary
  sudo /Library/TeX/texbin/tlmgr install varwidth
  sudo /Library/TeX/texbin/tlmgr install titlesec
  

Generate the user's documentation

Set the overall version (e.g. build number) in docs/source/conf.py

The OpenAPI yaml unresolved specifications are stored in the specs directory.

The naming of the specification are: PDS_APIs-{application}-{version}-swagger.yaml

For example: PDS_APIs-registry-1.0.0-swagger.yaml

To add a new specification or a new version of a specification:

1. Copy the OpenAPI Specification under the specs directory following the above filename convention.

2. Add a new entry for this file in docs/source/specifications.rst

3. In the file docs/source/conf.py add a section for the new specification in the redoc object.

4. Generate a PDF of the documentation.

   cd docs/
   make latexpdf
   cp build/latex/pdsapis.pdf build/html/_static
   

5. Generate the Sphinx documentation:

   pip install -e '.[dev]'
   
   sphinx-build -b html docs/source docs/build/html
   

Code of Conduct

All users and developers of the NASA-PDS software are expected to abide by our Code of Conduct. Please read this to ensure you understand the expectations of our community.

Development

To develop this project, use your favorite text editor, or an integrated development environment with Python support, such as PyCharm.

Contributing

For information on how to contribute to NASA-PDS codebases please take a look at our Contributing guidelines.

CI/CD

The template repository comes with our two "standard" CI/CD workflows, stable-cicd and unstable-cicd. The unstable build runs on any push to main (± ignoring changes to specific files) and the stable build runs on push of a release branch of the form release/<release version>. Both of these make use of our GitHub actions build step, Roundup. The unstable-cicd will generate (and constantly update) a SNAPSHOT release. If you haven't done a formal software release you will end up with a v0.0.0-SNAPSHOT release (see NASA-PDS/roundup-action#56 for specifics).

pds-api's Issues

Implement response format definition in API Service

Standardize management of empty attributes

choose between null, nan or not having the attribute or other options

Acceptance criteria:
Check that specification is clear about the management of empty fields or attributes, see:

1. https://github.com/NASA-PDS/pds-api/blob/master/docs/spec/pds-api-specification.md
2. and online https://nasa-pds.github.io/pds-api/

Perform trade study on various query language parsers to meet needs of proposed API Spec

DN Pilot API Project(s)

Update public PDS API Spec in Github

Define collection-level search parameters

Implement actual facet result in the response summary

Design and elicit requirements for PDS API Spec

Refine format conventions defined by the API WG

develop a jupyter notebook demo where a user can browse PDS archive from bundle to product data file

This requires:

• pick a use case
• ingest PDS4 archive with data files in pds-gamma registry
• make the python client work in jupyter notebook
• develop the full use case, with data access through http

Define initial set of intra-discipline (product-level) search scope

We need to come up with some intra-discipline (e.g. Imaging data) search parameters to include in the API.

Some questions:

• at what scope should we include this?
• should this just be custom query parameters (e.g. filter_name, filter_number, etc. == filter)
• for the parameters, should we define a new schema (e.g. custom/filter or search/filter ?)

Streamline testing of API server implementation

As a API developer, I want to be able to test that my API complies with the specification from a ready made test collection in POSTMAN.

Propose an acceptable syntax for PDS4 Xpath in json object

Acceptance criteria:
Check that specification is clear about the syntax of properties in JSON response, see:

1. https://github.com/NASA-PDS/pds-api/blob/master/docs/spec/pds-api-specification.md
2. and online https://nasa-pds.github.io/pds-api/
3. validate also in reference implementation https://pds-gamma.jpl.nasa.gov/api/swagger-ui.html

Refine response format conventions and parameter definition per #17

Initial Query Syntax Lexer Implementation

The demo API should support the query syntax described on https://github.com/NASA-PDS/pds-api/blob/master/docs/spec/pds-api-specification.md#query-syntax in parameter 'q'.

The wildcard (*, ?) do not need to be supported in this version.

Acceptance criteria:
Test simple comparisons and combination of comparisons as documented in https://nasa-pds.github.io/pds-api/#tag/collections

You can use demo API deployment available on https://pds-gamma.jpl.nasa.gov/api/swagger-ui.html or deploy your own as descrbed in pds-registry-app package documentation version 0.3.0.

PDS API extension convention for discipline-specific search engines

Define a PDS API extension model / convention for discipline-specific search engines

Implement EN query passing to DN search UI

Identify nodes and define scope for piloting pds-api-service adaptation

As a user, I want the API to be schema.org compliant

💪 Motivation

...so that I can easily integrate the PDS API with other APIs.

So that PDS datasets are viewed by google as datasets, see for example

📖 Additional Details

We want to be able to add a piece of javascript code in our HTML dataset landing page (bundle, collection, observational_products), which will call the registry-api with the lidvid of the current product as an argument and complete the schema.org json-ld description of the dataset (e.g. https://schema.org/Dataset) as described on this page https://developers.google.com/search/docs/appearance/structured-data/dataset#example from the registry-api response content.

The request will use a specific Accept header value application/ld+json in the content negociation.

⚖️ Acceptance Criteria

Given
When I perform
Then I expect

⚙️ Engineering Details

See introduction https://schema.org/docs/data-and-datasets.html

Bundle/Collection labels: https://schema.org/DataCatalog
Observational product labels: https://schema.org/Dataset
Data: https://schema.org/DataDownload

As a user, I want to be able to access documentation associated to a product description

or check it is available

Implement EN query of DN API service

Manage content negociation

The api end-point should answer in json or XML.

For XML response, the original PDS4 label should be provided

Document basic adaptation capabilities from pds-api-service

Deploy PDS API v0 (alpha) on pds-gamma test server

As a developer, I want a continuous deployment of the API available for testing

...so that we can utilize the PDS Client and test the API at "any time" against some deployed version with some data in it.

aka. CI/CD of the latest API server

Acceptance Criteria

Given I've made a change to the source code of the PDS API Service (ignoring docs)
When I perform a commit in Github
Then I expect to trigger a build and deployment to pds-gamma (and/or AWS if that is also reasonable)

Deploy PDS API v0 (alpha) for beta testing

Initial discipline/node-specific search parameter definition

Define initial prototype set of discipline/node-specific search parameters for API Spec

Propose PDS style for API documentation

For general documentation:
https://nasa-pds.github.io/pds-api/

Possibly for implementation documentation, for example:
https://pds-gamma.jpl.nasa.gov/api/swagger-ui.html

We want to keep the original style because user's are used to it since it comes from generic popular documentation generation but we want to show that it is a PDS documentation (at least logo) and provide links to the main web resource of PDS (app bar ?).

Improvements to Keyword Search for Weighting and Suggested Searches

Engineering Details

• Initial tweaks of ES query result weighting per NASA-PDS/registry-api-service#5
• Develop some simple docs for how we can perform some additional tweaks down the road per #93

Related Requirements

🦄 #99

Generate error when non existing fields are requested in field param

Currently the non existing field is returned in objects with value=null.

We should return error 400 (bad request) or 404 (non existing).

OGC EDR chose error 400, see https://github.com/opengeospatial/ogcapi-environmental-data-retrieval/blob/master/19-086.pdf

Assign and iterate over applicable API spec sections with API WG

Initial Federated API implementation

Core services of the API provided in the template. Basic infrastructure services will be built in

Acceptance criteria:

1. A basic implementation of the API is available open source
2. the implementation is publicly deployed online and documented

add lexer to registry api

Query passing and search integration design

Design how query passing will happen across the PDS both EN-to-DN and DN-to-DN

Deploy PDS API v0 (beta) on pds-gamma test server

[registry] Handle PDS Supplemental Metadata

How should PDS handle enriched / supplemental metadata?

• Product_Observational_Supplemental?
• Table data that is tagged as non-archival?
• Define new search/operations table data?
• Should this be ingested into the registry? or is this separate from registry and is for search specifically? separate "section" of response format to denote supplemental or additional metadata that is non-archival?

Motivation

...so that I can ingest supplemental metadata into the registry that is both archival (Product_Metadata_Supplemental) and non-archival (e.g. external databases) to make the information searchable and accessible.

Additional Details

Note: This ticket is the parent epic for the overarching design consideration, there are several sub-tickets for the individual data source implementations.

Some use cases we need to think about:

1. Use the PDS4 product Product_Metadata_Supplemental to revise data in products in the archive. NOTE: Need to check with RMS whether or not these should be considered archival.
2. Non-archival metadata - this is "augmented" or "supplemental" metadata that is generated after archiving that can help for search and/or improves the existing archive metadata. e.g. image tags indicating a crater exists in the image, or more detailed target information like Mars - Olympus Mons, or tracking service information like "here is this products DOI that was never registered)"
   a. should we just treat these the same as Product_Metadata_Supplemental? (depends if those are considered archival or not)
3. PDS3 databases - how do we integrate with databases that contain PDS3 data?
   a. should we just treat these the same as Product_Metadata_Supplemental? (depends if those are considered archival or not)

Acceptance Criteria

See sub-tickets for individual data source acceptance criteria

Engineering Details

Design thoughts

Should we just use Product_Metadata_Supplemental as the model for this and then augment for the non-archival data sources? - it seems like we should require some sort of archive-worthy definition of the data to be ingested. this could almost be a one-to-one translation of Product_Metadata_Supplemental, but just make it non-archival.

For the external databases, and initial workaround, we may also be just using a CSV for the time being: NASA-PDS/registry#51, and implement a MySQL plugin down the road since it would require some additional data modeling and configuration to connect to the database.

Manage field preselection in queries

As a user, I want to be able to select the fields I want in my search response of collections.

The management of null fields should be handled:

• when no fields are selected, all available fields are returned (no null fields)
• when fields are selected, all selected fields are returned. When they are not available they are set to a "null" value. This value should not be mis-interpreted for an original "null" in the pds model (to be detailed).

Acceptance criteria:
Check that specification is clear about the syntax to select fields in the JSON response, see:

1. https://github.com/NASA-PDS/pds-api/blob/master/docs/spec/pds-api-specification.md
2. and online https://nasa-pds.github.io/pds-api/
3. validate also in reference implementation https://github.com/NASA-PDS/registry-api-service deployed on https://pds-gamma.jpl.nasa.gov/api/swagger-ui.html

Identify a framework to generate python api client from a swagger file

Identify it and test it.

SwaggerHub CodeGen Server adaptation and integration how-tos

Document SwaggerHub CodeGen Server adaptation and integration how-tos

Update API Spec with refined response format conventions and field definitions

Define initial structure for response format conventions and parameter definition

• Refine format conventions defined by the API WG
• What are the default return fields?
• What are the required return fields?
• How do we split out "default" from "additional" metadata. We need to make sure the "additional" metadata fields returned do not come off as secondary citizens
• Update API Spec appropriately
• Implement in API Service

Design initial collection-level search API

Define default response fields

Acceptance criteria:
Check that the default fields are described in the specification:

1. https://github.com/NASA-PDS/pds-api/blob/master/docs/spec/pds-api-specification.md
2. and online https://nasa-pds.github.io/pds-api/

Convert PDS API Spec to Open API

Propose an initial server stub based on swagger output

The server stub will be automatically generated from swagger file so to be updated when swagger file is.

The implementation is java.

Define response formatting for default vs "additional" metadata fields

Define required response fields

Acceptance criteria:

Check that required fields are defined in the specification:

1. https://github.com/NASA-PDS/pds-api/blob/master/docs/spec/pds-api-specification.md
2. and online https://nasa-pds.github.io/pds-api/

initiate a proposal for the API WG and ENG team

Implement content negotiation

The api end-point should answer in json or XML.

For XML response, the original PDS4 label should be provided

Acceptance criteria:
Check that the API end-point support standard content negociation as describe here https://restfulapi.net/content-negotiation/ (especially header parameter Accept=application/json or application/xml or application/pds4+xml)

Add investigation area, mission information as external reference in API

Get the investigation area from the xml label, rewrite how target and instrument are read, get them also from the XML label

Acceptance criteria:
Check that target,instruments and investigation area are available in reference implementation https://github.com/NASA-PDS/registry-api-service and deployed on pds-gamma (https://pds-gamma.jpl.nasa.gov/api/swagger-ui.html)