ec-usgs / pywatershed Goto Github PK
View Code? Open in Web Editor NEWA hydrologic model in Python
Home Page: https://pywatershed.readthedocs.io
License: Creative Commons Zero v1.0 Universal
A hydrologic model in Python
Home Page: https://pywatershed.readthedocs.io
License: Creative Commons Zero v1.0 Universal
I'm trying to run something based on examples/02_prms_legacy_models.ipynb. I already have the variables prcp, tmax, and tmin in a single netcdf file that was written by Parker with his Bandit extraction code. I don't need to convert ascii CBH files into NCF files. I've been looking in the pywatershed code trying to figure out how to specify multiple variables that reside in one .nc but I can't figure it out. Is it possible to do this, or do I need to split them into three files with one variable in each?
The title says it all. Make sure our current, main users dont find that it breaks anything.
As part of introducing new processes from non-PRMS models, we need to first make PRMS/NHM processes work as we'd want the others to work. The following steps will (more or less) take us there
There are a variety of features that it would be nice for the Control class to implement:
This would be good to track additions as we go
https://github.com/pydata/xarray/blob/main/doc/whats-new.rst
produces
https://docs.xarray.dev/en/stable/whats-new.html
https://mypy.readthedocs.io/en/stable/
Do base classes as a start and their tests.
This may be ambitious for 0.1.3.
Coming to this repository with a curious eye as an outsider, it's not clear what's going on or why. Some kind of description of what this is in the README would be useful.
Add about section to the docs for version 1.0 to describe how pywatershed matches PRMS NHM configuration and how to perform the comparison.
pywatershed/pywatershed/base/data_model.py
Line 854 in cdf1a91
The functionality of this is not clear and needs a test to document it.
test all platforms
test both python 3.9 and python 3.10?
This release focuses on pywatershed reproducing PRMS5.2.1
This is an extended description repeating the milestone and allowing or additional discussion.
https://www.usgs.gov/products/software/software-management/types-software-review
add python 3.10 to the version testing matrix. consider dropping 3.8
This will save some significant time.
However
There's a few subfolders of dubious need.
I think documentation/ can stay. Not sure about the rest. @jdhughes-usgs
currently many data_model and DatasetDict tests are not carried out using netcdf4, but only xarray. this is because certain variable types (particularly time) needs extra code. I explored the code for how xarray solves this (using netcdf4 as a backend), but it is not obvious.
along these lines, the test_prms_param_separate.py does not separate parameters using netcdf4. Would be good to have that included.
Automate PyPi release as part of CI like done on FloPy
I've been running the example 02_prms_legacy_models.ipynb notebooks from a pywatershed installation on my windows 10 machine (not Whole Tale). This isn't necessarily a showstopper for me, but I want to report this error:
AssertionError Traceback (most recent call last)
Cell In[16], line 7
1 submodel = pws.Model(
2 submodel_processes,
3 control=control,
4 parameters=params,
5 )
----> 7 pws.analysis.ModelGraph(
8 submodel,
9 hide_variables=not show_params,
10 show_params=show_params,
11 process_colors=palette,
12 ).SVG(verbose=True, dpi=48)
File D:\PYWATERSHED\pywatershed\pywatershed\analysis\model_graph.py:120, in ModelGraph.SVG(self, verbose, dpi)
118 if self.graph is None:
119 self.build_graph()
--> 120 self.graph.write_svg(tmp_file, prog=["dot", f"-Gdpi={dpi}"])
121 if verbose:
122 print(f"Displaying SVG written to temp file: {tmp_file}")
File ~\AppData\Local\mambaforge\envs\pws\lib\site-packages\pydot.py:1743, in Dot.init..new_method(path, f, prog, encoding)
1739 def new_method(
1740 path, f=frmt, prog=self.prog,
1741 encoding=None):
1742 """Refer to docstring of method write.
"""
-> 1743 self.write(
1744 path, format=f, prog=prog,
1745 encoding=encoding)
File ~\AppData\Local\mambaforge\envs\pws\lib\site-packages\pydot.py:1828, in Dot.write(self, path, prog, format, encoding)
1826 f.write(s)
1827 else:
-> 1828 s = self.create(prog, format, encoding=encoding)
1829 with io.open(path, mode='wb') as f:
1830 f.write(s)
File ~\AppData\Local\mambaforge\envs\pws\lib\site-packages\pydot.py:1956, in Dot.create(self, prog, format, encoding)
1944 message = (
1945 '"{prog}" with args {arguments} returned code: {code}\n\n'
1946 'stdout, stderr:\n {out}\n{err}\n'
(...)
1952 err=stderr_data,
1953 )
1954 print(message)
-> 1956 assert process.returncode == 0, (
1957 '"{prog}" with args {arguments} returned code: {code}'.format(
1958 prog=prog,
1959 arguments=arguments,
1960 code=process.returncode,
1961 )
1962 )
1964 return stdout_data
AssertionError: "dot" with args ['-Tsvg', '-Gdpi=48', 'C:\Users\markstro\AppData\Local\Temp\1\tmprshir33o'] returned code: 3221225477
this is too hard-coded
require global metadata in the file saying "generic=specific"?
in absence of global metadata, have a default to the non-time dimension?
is there another way in parameters or control?
These variables are really booleans but we are treating them as ints to be consistent with PRMS. This is happening in a translation step in meta.py that I'm removing, so I'm changing the types to int in the metadata/variables.yaml. These should eventually be changed to boolean in the metadata and handled as such. this list may not be comprehensive.
diff --git a/pynhm/static/metadata/variables.yaml b/pynhm/static/metadata/variables.yaml
index 17e075c..fdd19b3 100644
--- a/pynhm/static/metadata/variables.yaml
+++ b/pynhm/static/metadata/variables.yaml
@@ -1636,7 +1636,7 @@ iasw:
on curve and maximum (1) or is on the defined curve (0)
dimensions:
0: nhru
- type: bool
+ type: int32
units: none
imperv_evap:
desc: Evaporation from impervious area for each HRU
@@ -1896,7 +1896,7 @@ lst:
the albedo curve (1) (albset_snm or albset_sna) otherwise (0)
dimensions:
0: nhru
- type: bool
+ type: int32
units: none
lwrad_net:
desc: Net long-wave radiation for each HRU
@@ -2109,7 +2109,7 @@ pptmix_nopack:
present on an HRU (1), otherwise (0)
dimensions:
0: nhru
- type: bool
+ type: int32
units: none
precip:
desc: Precipitation at each measurement station
allow subsetting along coords using numpy.where approach.
I'm running my NHM CONUS input files through the latest pywatershed develop version. I get the exception "AttributeError: 'PRMSSoilzone' object has no attribute '_adjust_parameters'" at pywatershed\hydrology\prms_soilzone.py:319 in _set_initial_conditions
I forgot to say that when I ran the much smaller Willamette model I didn't have this problem.
https://asv.readthedocs.io/en/stable/index.html
Implement performance import, some basic functionality, and NHM configurations.
Since there is an advantage to using PRMS control and parameter files for existing PRMS users. The output variables specified in the control file should be used to define the variables output to NetCDFs unless overridden using args in .initialize_netcdf(*args, output_vars=["rain",])
.
Should these files be in the develop distribution? Can't find them.
ImportError: cannot import name 'gis_files' from 'helpers' (C:\Users\jdickins\AppData\Local\mambaforge\envs\pywatershed\lib\site-packages\helpers_init_.py)
Currently NetCDF output can add significant wall clock time, especially when separate_files=False
. Performance needs to be improved.
Since pyproject.toml
has been setup to install required, lint, test, and optional dependencies a single development environment.yml
should be created.
environment.yml
should have all of the dependencies in pyproject.toml
environment.yml
should be in the root repo directory.environment.yml
files should be eliminated.*.txt
equivalents of environment.yml
are needed. Delete if not needed.use pyprms to do this
The Parameters object is read-only, so how do we edit its values for applications such as sensitivity analysis?
in PRMSSoilzone, the variable soil_moist is diagnostic, therefore it should not have a prior state that is tracked soil_moist_prev because it is not computed from it. Confusingly, I'd moved soil_moist_prev out of the .advance() but this caused an issue with PRMSRunoff not getting the correct soil_moist_prev. The fix is to remove soil_moist_prev from Soilzone and make PRMSRunoff get the prognostic inputs soil_lower_prev and soil_rechr_prev.
For "unit"/process tests in pywatershed against PRMS output, we used a variety of different approaches.
Failure is intermittent on Windows. Made some attempts to fix but did not resolve the issue. Does not seem to fail when run in serial but occasionally fails when run in parallel.
Notebooks exist in multiple places in the repo (evaluation, examples). Also it is unclear if all of these are necessary. In a recent PR (#164) several notebooks were out of date with recent commits.
Notebooks should also be run as part of CI to ensure that can be run with the current version of pywatershed.
-[X] Add sagehen unstructured HRUs #288
-[ ] Add sagehen gridded HRUs
-[ ] Adopt GSFLOW 2.3 as basis for regression tests. Use git submodules?
-[ ] Use recent flow accumulation work in pygsflow for cascades
When writing output variable netcdf files, I am trying to specify the output vars explicitly (using control.options), instead of writing all output var files.
control.options = control.options | {
"input_dir" : work_dir,
"budget_type" : None,
"verbose" : False,
"calc_method" : 'numba',
"netcdf_output_var_names": ['recharge']
}
But, it seems to not write "recharge" in this example.
I can still get all vars to write using:
multi_proc_model.initialize_netcdf(
output_dir=out_dir,
separate_files=True,
)
But, this takes a very very very long time. I'd rather fix the first block (runs faster). I'll paste the whole run script below.
Thanks
Eddie
import pathlib as pl
import time
import pywatershed
work_dir = pl.Path("./")
out_dir = work_dir / "output"
out_dir.mkdir(parents=True, exist_ok=True)
params = pywatershed.parameters.PrmsParameters.load_from_json(work_dir / "parameters.json")
control = pywatershed.Control.load(work_dir / "control.test")
control.options = control.options | {
"input_dir" : work_dir,
"budget_type" : None,
"verbose" : False,
"calc_method" : 'numba',
"netcdf_output_var_names": ['recharge']
}
multi_proc_model = pywatershed.Model(
[pywatershed.PRMSSolarGeometry,
pywatershed.PRMSAtmosphere,
pywatershed.PRMSCanopy,
pywatershed.PRMSSnow,
pywatershed.PRMSRunoff,
pywatershed.PRMSSoilzone,
pywatershed.PRMSGroundwater,
pywatershed.PRMSChannel],
control=control,
parameters=params
)
multi_proc_model.initialize_netcdf(
output_dir=out_dir,
separate_files=True,
)
sttime = time.time()
multi_proc_model.run(finalize=True)
print(f'That took {time.time()-sttime:.3f} looong seconds')
@jmccreight After meeting last week I wanted to document what we discussed and some of my thoughts. Hopefully we can use this to discuss and document possible changes to the control metadata.
Metadata was initially created from the published control variable tables and scraping the PRMS5 code.
<control>
<control_param name="basinOutON_OFF" version="5.0" deprecated="6.0">
<default>0</default>
<force_default>0</force_default>
<type>1</type>
<numvals>1</numvals>
<desc>Switch to specify whether or not basin summary output files are generated</desc>
<values type="flag">
<value name="0">off</value>
<value name="1">on</value>
</values>
<related_variables>
<variable name="basinOutVars"/>
<variable name="basinOut_freq"/>
</related_variables>
</control_param>
...
</control>
Each control variable has:
Key | Description | XML Element | XML Attribute | Allowed values |
---|---|---|---|---|
name | Name of the control variable | control_param | name | |
version | Version of PRMS where variable was introduced | control_param | version | |
deprecated | Optional version of PRMS when variable no longer supported | control_param | deprecated | |
default | Default value to use when one has not been specified | default | ||
force_default | Optional boolean indicating the default value should always be used | force_default | 0, 1 | |
type | Data type of the control variable values | type | 1, 2, 3, 4 | |
numvals | Number of values allowed for variable | numvals | ||
desc | Description of variable | desc | ||
values | Optional block for denoting accepted values of variable | values | ||
values->type | Type of valid values | values | type | flag, module, interval, method, parameter |
value | Description of valid value in values block | value | ||
value->name | A valid value | value | name | |
value->version | Version when a value option was added | value | version | |
related_variables | A group of variables that are associated with current variable | related_values | ||
variable->name | Name of control variable associated with current variable | variable | name |
type
description1: int32
2: float32
3: float64
4: string
values->type
descriptionflag
- Either boolean compatible values (1/0 = true/false, on/off, yes/no) or multiple values indicating behavior(s) to trigger
interval
- values indicating the time-series interval to use (e.g. 0 = daily)
method
- values indicating an action to perform or module to act on
module
- values representing modules to use in simulation code
parameter
- values denoting one or more parameters to effect
context
key which would have values of scalar
, array
, etc
numvals
?start_time
and end_time
which are really datetime values but stored in control file as an array of integersLegacy XML structures are converted to a dictionary and written to YAML.
basinOutON_OFF:
default: 0
desc: Switch to specify whether or not basin summary output files are generated
(0=no; 1=yes)
numvals: 1
related_variables:
- basinOutVars
- basinOut_freq
type: 1
values:
0: 'off'
1: 'on'
values_type: flag
version
usage could be changed to also indicate a range of versions. If we did this then the deprecated
key would not be necessary.
version=">=5.0.0 & <5.2.0"
context
key with possible values scalar
, array
, etctype
-> datatype
values
-> valid_values
values_type
-> valid_values_type
int32: 1
float32: 2
float64: 3
string: 4
datetime:
Currently, from numpy.distutils.misc_util.Configuration
is used to build Fortran components. This approach is currently deprecated. When running
python -m build
The following is reported:
`numpy.distutils` is deprecated since NumPy 1.23.0, as a result
of the deprecation of `distutils` itself. It will be removed for
Python >= 3.12. For older Python versions it will remain present.
It is recommended to use `setuptools < 60.0` for those Python versions.
For more details, see:
https://numpy.org/devdocs/reference/distutils_status_migration.html
TODO:
Also, since it seems we are getting ~equivalent performance using numba maybe we should
Currently reused parameter instances cause errors in the channel network.
My ethos is that parameters should not be edited in the dark, hidden corners of the code. They should be edited as preprocessing so that they can be provenanced with the results.
My prefered approach to this problem is currently to freeze/imutablize parameter data. Two approaches
https://adamj.eu/tech/2022/01/05/how-to-make-immutable-dict-in-python/
https://pypi.org/project/frozendict/
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.