usace / cumulus

Cumulus project issue tracking and project planning

usace

US Army Corps of Engineers

cumulus's Issues

Merge cumulus-api and cumulus-ui into single repository

There are now many different services in the cumulus-api repository. Migrate the user interface into the cumulus-api repository and move repository USACE/cumulus-api --> USACE/cumulus.

automate loading local database

automate local db container to load schema/seed data from scratch on start

Productfile Versioning

CI/CD: Separate Github Actions

All github actions jobs may not be required, depending upon which file in the repository changes. Separate github actions so only necessary build and deploy steps are triggered. For example, do not rebuild packager container if only a file in the REST API was changed.

Add Support for MyWatersheds

Add new API endpoints for addition/modification/deletion of user's watersheds (or watershed favorites). Adjust UI to connect to new endpoints.

Basic GET endpoints for `area_group` and `area`

• Endpoint for listing area_group for a given watershed: /watershed/:watershed_id/area_groups
• Endpoint for listing areas for a given area_group: /watershed/:watershed_id/area_groups/:area_group_id/areas

add watershed "oahe to gavins" for NWDM

Update API Root Url

Change API Root URL to: https://water-api.rsgis.dev/cumulus/v1

Airflow DAG for ncep_rtma_ru_anl_airtemp

DAG to download raw files from NCEP and write to S3

Migrate Data Downloads to Airflow

Add support for MyDownloads

Adjust database tables, add/modify API end points to support user's downloads for dashboard. Adjust UI to display new endpoint data.

Upgrade TailwindCSS to v2.0

https://blog.tailwindcss.com/tailwindcss-v2

WPC 2.5 Kilometer Real Time Data Acquisition

DAG to download raw PRISM files and write to S3

• tmin_early
• tmax_early
• ppt_early

github workflow for new api container

DSS Packager: Grid Projection

Grid projection not defined properly. Grid info defined as Albers does not show proper meta data.

LTIA98 Air Temperature Grids

Packager; pydsstools pip install location

Change pip install location of pydsstools to USACE/pydsstools develop branch in this requirements.txt file:
https://github.com/USACE/cumulus-api/blob/develop/async_packager/packager/requirements.txt

API Containers: Allow Actions Manual Builds (workflow_dispatch)

Statistics; Refactor API to Enable/Disable Statistics for Area Groups

Packager _SQS Environment Variables

Environment Variables for Overriding Usual AWS Credentials for SQS Configuration are:

AWS_ACCESS_KEY_ID_SQS
AWS_SECRET_ACCESS_KEY_SQS
AWS_REGION_SQS

If these variables are unset, set them to the default AWS environment variables for user credentials. Specifically:

AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY
AWS_REGION

This will allow these values in development using docker-compose:

AWS_ACCESS_KEY_ID_SQS=x
AWS_SECRET_ACCESS_KEY_SQS=x
AWS_REGION_SQS=elasticmq

When deployed in AWS, the variables suffixed with _SQS will be unset and the configuration variables will be set to the AWS usual variables set in the environment (without the _SQS suffix).

Download Capability from Web UI

Allow users to create DSS Downloads from the web User Interface. Implement as a modal on /dashboard route, near recent downloads component.

Repository Refactor

• move docker-compose.yml to repository root; associated path changes
• move go API from folder "root" to "api". make it a separable module
• make async_listener a go module
• Remove go module files (go.mod) from repository root
• Containerize api so it can be run in the same docker network as database, listener, etc. To date it has been run in the vscode debugger on the native os. This move allows other containers - specifically async_packager and async_statistics - to easily interact with the database via the http API using http://api:3030/.

Initial Basin page UI mockup

fix profile UI style to match dashboard, ws, download

MBRFC Hourly Air Temperature Forecast

• Air temperature product provided to Omaha District
• 16 days of hourly air temperature forecast. Provided to district by RFC at present via LDM

Product File Seed Data

Additional productfile seed data is needed for local testing of downloads and user interface. Create local script to generate database inserts. Script will use the AWS api to fetch products and product files. An adjustable time window will be used in local script. Script will output the sql file into the database folder for auto consumption on db container start.

Basin Groups Support (Columbia River Basin, Major Tributary, Subbasin Concept)

🚨 Planning in Progress

Tables:
basin (columbia river)
subbasin (all subbasins, all major trib)
subbasin_group ("subbasin", "major trib")
subbasin_group_subbasins (many-to-many)

Initial GET Endpoints
List Basin Groups for Given Basin
GET /cumulus/basins/columbia-river/subbasin_groups
- Payload: [
{id: uuid, name: 'Subbasins', slug: 'subbasins'},
{id: uuid, name: 'Major Trib', slug: 'major-trib'},
{id: uuid, name: 'Basin', slug: 'basin'}
]

Get Basin Group
GET /cumulus/basins/columbia-river/basin_groups/subbasins
- Payload: { id: 'uuid', name: 'Subbasin', slug: 'subbasin', basins: [{id: uuid, name: "", slug: ""}, {}, {}]}

Add Minio Container as a S3 Standin for Local Development

AirFlow DAG to download PRISM Stable tmin, tmax and ppt from 1981 to present or (last available)

Bugfix; DSS Dpart 2400 --> 0000

DSS dPart time should be "0000" for midnight (not 2400) unless data_type is INST-VAL.

[PRODUCT]: Soil Moisture - NLDAS and SMAP

Add GH Action for CI/CD Listener Container

• Add Github Action to create container
• Create target repository for image upload
• Role adjustments to allow image upload for CI user

initial dashboard and watershed UI mockup

NGDG_ltia98_historic_temperature_downloads

Download available archive which looks to go back to Jan 2011.

Automate Functional API Tests With API Docker Container

Functional tests are written using postman. Automate a way to run the functional tests against the API running in the docker-compose stack locally. This puts the foundation in place to run the tests automated on code commit (to be completed as part of a separate issue).

Packager; SQS Payload Exceeding Limits

As the data time window increases or the number of products increases, the payload length sent to SQS also increases. Refactor messaging so a UUID for a download request is sent to the message queue. Then implement a single query of the API in the worker.

Watershed Endpoint; Remove /basins endpoint

• Add /watersheds endpoints; include bbox on watersheds
• Store geometry field on watersheds
• Remove /basins endpoint and fix associated ripple effects

SNODAS Interpolated Products

LTIA98 Precipitation Grids

pydsstools; Origin (lower-left) always 0, 0

Investigating solution to DSS Grid creation with pydsstools always results in the grid origin (lower-left) is always at zero. Focusing on pydsstools C-bindings and gridInfo().

Watersheds and Area Groups: New Tables

Add new table structure to support watershed and area groups concept.

Create, Read, Update, Delete Watersheds

RESTful API Endpoints for Watersheds

fix dpart dss download to 2400

dpart should be 2400 instead of 0000 at midnight for inst grids (like airtemp)

Add Workflow / Pipeline Framework for Data Processing (apache/airflow)

Add apache airflow for data processing pipelines.

Add "wpc_qpf_2p5km" to db products, remove "ncep_rtma_ru_ges_airtemp"

"ncep_rtma_ru_ges_airtemp" had no productfiles in dev. One download record and download_product record was present. They were removed from dev.

CreateDownload Does Not Include All Products Requested

Fix Regression Tests

Fix outdated regression tests. Ensure the regression test suite runs from top to bottom. This will serve as the baseline test suite to be run before merge to develop branch. Tests should be expanded / improved as new capabilities and endpoints are added.

Download UI Mockup

Deploy async_listener to environment: development