nasa / opera-sds-int

Hi 👋, welcome to the NASA org on github.com!

Github.com/nasa has one of the largest collections of NASA open-source code repositories. Members of the NASA org can find instructions for github.com/nasa in http://nasa.github.io/.

🔭 Additional open-source code repositories resides in a variety of locations other than github.com/nasa. To discover code across all of these locations, we suggest you use code.nasa.gov & software.nasa.gov. These are two different sites holding metadata that describe code projects. Any code released through the NASA Software Release Authority process should be cataloged on those sites.

Code.nasa.gov —

Is a page with short descriptions of all of NASA's open-source code. Code.nasa.gov feeds into code.gov, which covers open-source and government-source code from many different U.S. governmental agencies. To assist in discovery, code projects described on code.nasa.gov have both human and A.I.-generated tags. These can be useful for finding related code projects.

software.nasa.gov —

Contains metadata descriptions for all code projects in code.nasa.gov as well as government-source code projects only sharable with other government agencies. It is part of the large https://technology.nasa.gov/ that also includes patents and spinoffs. To help discoverability, software.nasa.gov puts each code project into one fo the following categories: Business Systems and Project Management, System Testing, Operations, Design and Integration Tools, Vehicle Management (Space/Air/Ground), Data Servers Processing and Handling, Propulsion, Structures and Mechanisms, Crew and Life Support, Data and Image Processing, Materials and Processes, Electronics and Electrical Power, Environmental Science (Earth, Air, Space, Exoplanet), Autonomous Systems, and Aeronautics.

NOTE - PROFILE READMES CURRENTLY DON'T WORK FOR ORG PROFILES ONLY USER PROFILES :(

https://github.community/t/readme-for-organization-front-page/2920

opera-sds-int's Issues

Metrics collection

Brief Description

We need methods to capture some metrics to support our load testing goals. In some cases, there may be existing tools we can leverage already available in the PCM or by TPS tools, but in other cases, we may need to write custom scripts for metric collection.

Expected Behavior

• Capability to capture the following metrics:
  • Accumulated size (in bytes) of a given AWS S3 bucket over a given time frequency (down to minutes)
  • Throughput (in bytes/sec) of a given AWS S3 bucket over a given time frequency (down to minutes)
  • Elasticsearch statistics (num docs, query time, etc.) for a given index over a given time frequency (down to minutes)
  • Elasticsearch instance health over time (RAM, storage)
  • PCM queue sizes (QUEUED / PENDING jobs especially) over a given time frequency (down to minutes)
  • AWS EC2 spot errors (insufficient capacity/terminations)
• Capability to generate a CSV with metric values over time to support plotting

Current Behavior

The above have to be manually calculated by hand currently.

Suggested Ideas on Resolution

There are likely tools already available to capture all the above. Likely, we just need to create a script which can obtain the information above and post into a CSV file for easy plotting.

Some suggested resources to evaluate:

• https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-health.html
• https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-stats.html
• https://www.sumologic.com/blog/monitor-aws-s3-metrics/
• https://docs.aws.amazon.com/AmazonS3/latest/userguide/cloudwatch-monitoring.html
• HySDS UI -> RabbitMQ describes number of queued items (for a given queue) within PCM
• etc.

OPERA SDS has to update the NASA A&A process to obtain the ATO.

Steps:

1. Access the following link :
   https://cae-testrail.jpl.nasa.gov/testrail/index.php?/tests/view/3989252&group_by=cases:title&group_order=asc&group_id=-1

2. Update the test results accordingly

OPERA-SDS-VnV-8: Verify automated product generation

Requirement: "For each PGE (from current and previous releases), verify that uploading the proper inputs to the designated Input Storage Location will trigger a succesfull PGE execution, followed by PCM moving all generated output to the designated Output Storage location. Includes inspecting the job logs to verify successful job execution.

Inspect the output location of a PGE execution to verify that it contains files with quality metrics information, with fields specific to each PGE

Inspect the latency metrics (one for production time, and one for retrieval time for FWD processing) reports generated by the PCM system, and ensure that they contain all relevant information to asses whether generation of each OPERA products meets the product latency requirement for that product.

Inspect the accountability metrics generated by the PCM system, and verify they contain all neecssary information to asses that all expected output products are generated, for each PGE."

VnV-03: opera-sds/repositories landing page supposed to open source

Steps to Reproduce:
Open page: https://github.com/orgs/nasa/teams/opera-sds/repositories
you will notice this page requires login & does not show public:

Expected Result: Landing page need to be open sourced and does not require login to access the repositories: Please see the following link for the test results:

https://cae-testrail.jpl.nasa.gov/testrail/index.php?/tests/view/3989253&group_by=cases:title&group_order=asc&group_id=-1

OPERA-SDSVnV-13_2:Functional validation of the PGE for L2_RTC_S1 products

While testing noted multiple issues with the runconfig file:

1. I do not find the location to update:
   landcover_file: input_dir/TBD.tif
   worldcover_file: input_dir/TBD.tif

2. I have executed script with ignoring those options. getting the following errors:

hysdsops@opera-int-mozart-pop1:~/mozart/ops/opera-pcm/tools/OPERA-SDSVnV-13_2$ docker run --rm -u $UID:$(id -g) -v ${WORKING_DIR}/expected_output_dir:/home/conda/expected_output_dir:ro -v ${WORKING_DIR}/runconfig:/home/conda/runconfig:ro -v ${WORKING_DIR}/input_dir:/home/conda/input_dir:ro -v ${WORKING_DIR}/output_dir:/home/conda/output_dir -i --tty opera_pge/rtc_s1:2.0.0-er.5.0 --file /home/conda/runconfig/rtc_s1_sample_runconfig-v2.0.0-er.5.0.yaml
Running preprocessor for RtcS1PreProcessorMixin
Traceback (most recent call last):
File "/home/rtc_user/opera/pge/base/base_pge.py", line 96, in _validate_runconfig
self.runconfig.validate()
File "/home/rtc_user/opera/pge/base/runconfig.py", line 142, in validate
yamale.validate(pge_schema, runconfig_data, strict=strict_mode)
File "/home/rtc_user/miniconda3/lib/python3.9/site-packages/yamale/yamale.py", line 43, in validate
raise YamaleError(results)
yamale.yamale_error.YamaleError: Error validating data '/home/conda/runconfig/rtc_s1_sample_runconfig-v2.0.0-er.5.0.yaml' with schema '/home/rtc_user/opera/pge/base/schema/base_pge_schema.yaml'
RunConfig.Groups.SAS.runconfig.groups.input_file_group.burst_id: Length of [] is less than 1

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/rtc_user/opera/scripts/pge_main.py", line 187, in
pge_main()
File "/home/rtc_user/opera/scripts/pge_main.py", line 183, in pge_main
pge_start(run_config_filename)
File "/home/rtc_user/opera/scripts/pge_main.py", line 157, in pge_start
pge.run()
File "/home/rtc_user/opera/pge/base/base_pge.py", line 740, in run
self.run_preprocessor(**kwargs)
File "/home/rtc_user/opera/pge/rtc_s1/rtc_s1_pge.py", line 56, in run_preprocessor
super().run_preprocessor(**kwargs)
File "/home/rtc_user/opera/pge/base/base_pge.py", line 179, in run_preprocessor
self._validate_runconfig()
File "/home/rtc_user/opera/pge/base/base_pge.py", line 105, in _validate_runconfig
self.logger.critical(
File "/home/rtc_user/opera/util/logger.py", line 407, in critical
raise RuntimeError(description)
RuntimeError: Validation of RunConfig file /home/conda/runconfig/rtc_s1_sample_runconfig-v2.0.0-er.5.0.yaml failed, reason(s):
Error validating data '/home/conda/runconfig/rtc_s1_sample_runconfig-v2.0.0-er.5.0.yaml' with schema '/home/rtc_user/opera/pge/base/schema/base_pge_schema.yaml'
RunConfig.Groups.SAS.runconfig.groups.input_file_group.burst_id: Length of [] is less than 1
hysdsops@opera-int-mozart-pop1:~/mozart/ops/opera-pcm/tools/OPERA-SDSVnV-13_2$

Please see the ticket for details:
https://cae-testrail.jpl.nasa.gov/testrail/index.php?/tests/view/3996156&group_by=cases:title&group_order=asc&group_id=-1

OPERA-SDS-VnV-1: "Verify Operations begin as planned after CalVal"

Requirement: "Demonstrate that system operations will begin within 1 month of the completion of product validation, for each major release"

To do:
o Record the data when CalVal ends (for each PGE)
o Record the data when Operations begin (for the same PGE)

Verify that the difference is <= 1 month

OPERA-SDS-VnV_19: Ingest Dynamic Ancillary Data

Verification activity:
"Demonstrate that the PCM system can successfully retrieve and ingest dynamic ancillary files needed for PGE execution."

OPERA-SDSVnV-16: Verify archival of DSWX_HLS data products

L3 Requirement:
Verify that the standard data products are available to designated DAACS as defined in ICD. Use the CMR query service to locate the products of interest, and Cumulus data services to download the data.Execute these tests for each OPERA data products, and for multiple time intervals and geographic selections.

Verify that the SDS adheres to the external interfaces with the DAACs as specified in the ICD agreement, specifically:
o The SDS uses CNM ("Cloud Notification Mechanism") to notify the target DAAC that new output prodcuts are available for archiving

Verify that the DAAC metadata is delivered to the DAAC with the product.

TestRail case: https://cae-testrail.jpl.nasa.gov/testrail/index.php?/tests/view/3678197&group_by=cases:section_id&group_order=asc&group_id=69570

OPERA-SDS-VnV-18: Ingest Static Ancillary Data

Requirement: "Demonstrate the that PCM system can successfully ingest ancillary files "offline", and use these files during operations."

Things to do:
o Verify that static data for DEMs and Land Cover is stored on S3 buckets
o Submit a PGE via standard procedure
o Verify that the PGE uses static data by looking at the RunConfig and job log file

Load testing launch script

Brief Description

In order to kick off load testing (as well as future I&T automated tests) we'll need a "starter script" that should kick off the first step in our OPERA SDS PCM processing pipeline: querying the input DAACs for products.

Expected Behavior

• The general idea is to keep things simple and leverage a crontab file that lists all the possible queries for the PCM to invoke the DAAC's CMR for products to begin downloading/ingesting/processing
• New wrapper script that generates a crontab listing of scheduled times to invoke daac_data_subscriber_query.sh based on a configuration file that maps to the expected usage of the daac_data_subscriber_query.sh script (see below)

Current Behavior

• Working daac_data_subscriber_query.sh script
• @LucaCinquini's initial brainstorming on this idea, with a sample script he wrote

Suggested Ideas on Resolution

• Work with @maseca to ensure you can call the underlying daac_data_subscriber_query.sh script correctly, and @riverma on how this relates to the load test.

Release 1 I&T Report

We should put together the I&T/V&V Report for Release 1. This will include updating the VnV Matrix for R1 testing.

Updating the VnV Matrix for R1:

• GDrive link to a working version: OPERA SDS L3 VnV Matrix for R1 2022-06-27.xlsx
  • This copy is distinct from the spreadsheet we will use to track R2 testing activities.
• Update columns H-N for each of the VnV Activities exercised for R1.

What the report needs to contain (can be a doc, or slides):

• Identify the version(s) used for testing.
• summary table of the VnV Matrix
• Table with the L3 SDS Requirements addressed by R1, and their status (verified/closed, or unverified/open, etc.)
• Summary listing of any issues encountered during testing. Note which have already been addressed & closed, and which (if any) are still open.
• If there are any un-verified / open requirements, there should be a description of the issue(s) preventing closure, and a plan for how we will address them (and ultimately close the req.).

OPERA-SDS-VnV-2: Verify compliance with JPL and NASA Cybersecurity requirements

Requirement: "Verify that OPERA SDS has passed the NASA A&A process to obtain the ATO (Authorization To Operate)."

To do:
Simply record the data when ATO is given to OPERA, and provide a link to the official document. The ISO will be notified and will notify everybody else.

OPERA-SDSVnV-9_1: Manually validate PCM can Retrieve and download of HLS Data Products for the SHORT_NAME= HLSS30 & HLSL30

Demonstrate that the PCM system can successfully query CMR for available input products for all PGEs (from current and previous releases), and download them from LP.DAAC to the SDS Input Storage Location (ISL). Do this for different time range selections, including the latest products (to support FWD mode) and some time interval in the past (to support "catch-up" mode).

Verify that the SDS adheres to the external interfaces with the DAACs as specified in the ICD agreement, specifically:
o The SDS identifies its input products by querying the ESDIS CMR (Common Metadata Registry)
o The SDS downloads its input data products using the Cumulus download service deployed at the source DAAC

TestRail link:
https://cae-testrail.jpl.nasa.gov/testrail/index.php?/tests/view/3351557&group_by=cases:section_id&group_order=asc&group_id=69570

OPERA-SDS-VnV-21: Product Storage

VnV Activity:
"Verify that the system can retain files (input products, output products, and dynamic ancillary data) in SDS storage. The duration of storage should be configurable in the system (without need for redeploying). Verify that files are purged at the end of their duration."

OPERA-SDSVnV-13_3:Functional validation of the PGE for L2_CSLC_S1 products

Issue: PGE execution is throwing an error for the L2_CSLC_S1:

Please see the details here:

https://cae-testrail.jpl.nasa.gov/testrail/index.php?/tests/view/3996170&group_by=cases:title&group_order=asc&group_id=-1

OPERA-SDS-VnV-11: Verify SDS can recover from disruptions

Split this into multiple tickets depending on what disruption is simulated

OPERA-SDSVnV-7: Verify that generation of data products meets projects requirements

L3 requirement:
Inspect the data products generated by each PGE to verify that they cover the spatial and temporal extent specified in the project data generation table. Also, capture PCM execution metrics to verify product accountability (i.e. that all expected products are actually generated) and latency (i.e. that products are generated within the required time).

Update SDSVnV-08_1 to better reflect what testers see

The instructions in step 24 should be updated to better reflect what the tester will actually see when conducting the test. The instructions look like they are left-over from Release 1, and didn't get updated.

Update VnV Matrix for R2 Testing

This ticket is for updating the VnV Matrix for R2 Testing.

• GDrive link to a working version: OPERA SDS L3 VnV Matrix for R2 2022-10-17.xlsx
  • This copy is distinct from the spreadsheet we are using to track R1 testing activities.
• Ensure that each requirement allocated to R2 is sufficiently tested by the Verification Activities. If any are not, then new Verification Activities may need to be created.

OPERA-SDS-VnV-10: Verify the interface for control of the operational system

Verification Activity:

Verify that the interface may allow for control of the operational system, including:

• current processing load / activity
• identification of problems / issues in the system
• ability to halt or stop a given process
• monitor the health of the SDS
• ability to monitor on a per-product basis
• ability to initiate on-demand jobs
• (possibly more)

OPERA-SDS-VnV-14: Verify archival of DSWX_HLS data product documentation

Requirement: "Verify that all relevant information for the DSWX_HLS data product (description of science algorithm, validation datasets, test results, etc.) is available from the same DAAC where the data products are archived."

To Do:
o Define list of all documents that need to be archived at the DAACs- the DAAC that will host the data products

• product spec
• ATBD (Algorithm Theoretical Basis Document)
• SAS design?
• ICS?
  o Determine where the DAACs store these documents (look at existing missions)

OPERA-SDSVnV-13: Functional validation of PGE for DSWX_HLS products

L3 requirement:
Verify that the PGE for generation of L3_DSWX_HLS products, wrapping the final version of the corresponding SAS algorithm, is delivered to PCM. Test that the PGE can be successfully executed using the RunConfig file specified in the PGE/PCM ICS.

Verify that all products generated by the L3_DSWX_HLS PGE contain a version number in their file name. Verify that the version number is incremented if and only the checksum of the output data product is changed. In other words, generating the same product twice with exactly the same software should NOT result in a version increment.

Verify that the DSWX_HLS PGE is accompanied by a corresponding Interface Specification Document (ICS) which should contain the complete information needed by the PCM system to invoke and monitor the PGE, including:
o The input and ancillary data sources
o The format of the RunConfig file that encapsulates all the iinput parameters
o The set of EC2 machines that the PGE should be running on
o The success and error status code
o Any other relevant information

Inspect the output location of a PGE execution to verify that it contains files with quality metrics information, with fields specific to each PGE

Verify that DAAC Metadata is correctly produced for DWSX_HLS products, and that this is done for each DSWX_HLS product.

Inspect the data products, log files and metadata generated by the PGE and verify that all timestamps use the Universal Time Coordinate (UTC) and include the "Z" to denote the Greenwich time zone.

TestRail link: https://cae-testrail.jpl.nasa.gov/testrail/index.php?/tests/view/3351586&group_by=cases:section_id&group_order=asc&group_id=69570

OPERA-SDS-VnV-12: Verify the generation rate for the L3_DSWX_HLS product

OPERA-SDS-VnV-17: Timed test for retrieval of input products

VnV Activity:
"Verify that the SDS can retrieve and ingest input products for all available PGEs (for current and previous releases) automatically within the time allotted by OPERA-SDS-44 and OPERA-SDS-45.
"

OPERA-SDSVnV-6: Verify deployment of the SDS system to multiple AWS environments ("venues").

Verify that the SDS software stack can be deployed and operated in multiple AWS environments (aka "venues"): DEV for developers, I&T for Integration and Testing, CalVal for product validation, and OPS for operations.

TestRail link: https://cae-testrail.jpl.nasa.gov/testrail/index.php?/tests/view/3718320&group_by=cases:section_id&group_order=asc&group_id=69570

OPERA-SDSVnV-3: Verify the SDS software is Open Source

L3 Requirement:
Verify that the SDS software repositories are hosted on the public GitHub, and that other tools (forums, tickets, etc.) are available for Open Source development.
TestRail case: https://cae-testrail.jpl.nasa.gov/testrail/index.php?/tests/view/3551351&group_by=cases:section_id&group_order=asc&group_id=69570

OPERA-SDSVnV-9_2: Automated retrieval and Ingestion of HLS products

Demonstrate that the PCM system can successfully query CMR for available input products for all PGEs (from current and previous releases), and download them from LP.DAAC to the SDS Input Storage Location (ISL). Do this for different time range selections, including the latest products (to support FWD mode) and some time interval in the past (to support "catch-up" mode).

Verify that the SDS adheres to the external interfaces with the DAACs as specified in the ICD agreement, specifically:
o The SDS identifies its input products by querying the ESDIS CMR (Common Metadata Registry)
o The SDS downloads its input data products using the Cumulus download service deployed at the source DAAC

TestRail link: https://cae-testrail.jpl.nasa.gov/testrail/index.php?/tests/view/3718400&group_by=cases:section_id&group_order=asc&group_id=69570