jetilton / cwmspy

cwmspy

Wrapper around the HEC CWMS api

Visit the documentation for full description.

Installation

pip install git+https://github.com/jetilton/cwmspy.git

You will need to navigate to the package location and create a .env file with the appropriate user, password, host and service name to automatically connect to the CWMS Oracle database. You will also need to have followed the instructions for the python package cx_Oracle.

Usage example

from cwmspy import CWMS 

cwms = CWMS()
cwms.connect()
df = cwms.retrieve_ts(p_cwms_ts_id='Some.Fully.Qualified.Ts.Id', start_time='2019/1/1', end_time='2019/9/1', return_df=True)

df.head()

            date_time  value  quality_code
0 2018-12-31 23:00:00    NaN             5
1 2019-01-01 00:00:00    0.0             0
2 2019-01-01 01:00:00    0.0             0
3 2019-01-01 02:00:00    0.0             0
4 2019-01-01 03:00:00    0.0             0

Visit the documentation for more examples.

Meta

Jeff Tilton – [email protected]

https://github.com/jetilton/cwmspy

Contributing

1. Fork it (https://github.com/jetilton/cwmspy/fork)
2. Create your feature branch (git checkout -b feature/fooBar)
3. Commit your changes (git commit -am 'Add some fooBar')
4. Push to the branch (git push origin feature/fooBar)
5. Create a new Pull Request

cwmspy's Issues

Add a retrieve_period_of_record

Currently there is only the ability to get a time window, it would be nice to just be able to pull a period of record (POR) for a given time series.

THis would be done in the _Extra class using the get_min_date/get_max_date, which also need to be implemented.

cwms has inconsistent time inputs

Should figure out way to consistently accept input times. i.e. 01-01-2020 rather than both 01-01-2020 & 01/01/2020

Either accept both inputs or convert one input.

retrieve_ts accepts only 01/01/2020 style

retrieve_time_series and store_by_df accepts only 01-01-2020 style

Store_by_df

Should be a method to store by passing a pandas dataframe

store_by_df fails when there is missing data

store_by_df should be able to handle when data is missing within the timestamp. Need to create a minimal example to show current behavior and then expected behavior.

Delte_by_df

should be able to pass a pandas data frame with columns date_time, ts_id, value, that deletes all values in df

store_ts

store_ts should be able to:

• convert times to UTC before storing
• accept either string dates formatted "%Y/%m/%d" or datetime objects
• Assume 0 for all quality codes if p_qualities = None

Current behavior expects times in UTC and only accepts datetime objects

Need a restore pathname method using Flashback query

People make mistakes, need a way to bring data back within a reasonbale time frame. Oracle has a flashback feature that you can get

You can make queries against a flashback table. May need to look into configuring the tablespace.

current store_by_ has problems on some configurations

when "only_add_different=True"

cwms.store_by_df(transform_df, "UTC", only_add_different= False) fixes issue

retrieve_time_series has problems or fails with retrieving null values.

Length mismatch: Expected axis has 1 elements, new values have 3 elements

Try/catch fixes temporarily

Add tests for not storing/deleting protected values

Need to add tests for not storing/deleting protected values

• delete_by_df
• delete_ts_window
• store_ts
• store_by_df

Option for multi node connection

A multi-node connection would allow for methods associated with comparing different nodes.

Examples in docstrings

All methods should have examples in doctest format

delete_by_df only allows for 1000 value df.

Possibly add a for loop so delete_by_df handles 1000+ data frames

Add Retrieve_ts_multi

Retrieves time series data for multiple time series
api