uw-hydro / pysumma

Python module for managing SUMMA simulations

pysumma

pysumma Links & Badges
pysumma Tutorial
pysumma Documentation
Travis-CI Build

pysumma is a Python wrapper for manipulating, running, managing, and analyzing of SUMMA (Structure for Unifying Multiple Modeling Alternatives)

• SUMMA web site at UCAR

pysumma provides methods for:

• Running SUMMA
• Modifying SUMMA input files
• Automatically parallelizing distributed and sensitivity analysis type experiments
• Calibration via OSTRICH
• Visualizing output

Installation

pysumma can be installed from either conda or pip. Installation using conda is preferred, because it will also install a compatible version of SUMMA.

To install via conda use:

conda install -c conda-forge pysumma

To install via pip use:

pip install pysumma

Dependencies

A conda environment is available for management of pysumma's dependencies. You can create your own environment from this file by running:

conda env create -f environment.yml

Then, you can activate this environment with conda activate pysumma. Before installing pysumma into this environment you may also wish to install it as a kernel in your Jupyter environments. This can be accomplished by running:

python -m ipykernel install --user --name=pysumma

With this environment active you can install pysumma this environment with the instructions below.

Installing pysumma from source

Installing pysumma from source can be useful for developing new features. This can be accomplished by running:

git clone https://github.com/UW-Hydro/pysumma.git
cd pysumma
python setup.py develop

Additional SUMMA References

• Document : SUMMA documentation is available online and remains a work in progress.
• Source Code : NCAR github

Bugs

Our issue tracker is at https://github.com/UW-Hydro/pysumma/issues. Please report any bugs that you find. Or, even better, fork the repository on GitHub and create a pull request. All changes are welcome, big or small, and we will help you make the pull request if you are new to git (just ask on the issue).

How to run pySUMMA on HydroShare

(Link YouTube: https://www.youtube.com/watch?v=pL-LNd474Tw)

1. log in HydoShare(https://www.hydroshare.org)
2. Start CUAHSI JupyterHub from APPS menu on HydroShare(https://www.hydroshare.org/apps/)
3. Open "Welcome.ipynb" to download pySUMMA resources from HydroShare

• run the code "1.How to connect with HydroShare"
• change the code from "resid = os.environ['HS_RES_ID']" to "resid = 'c1bb4a12bff44bf08c5958cba7947348'".

4. You can see the list of Jupyter Notebooks and click one of Jupyter Notebook.
5. Run one of Jupyter Notebooks.

pysumma's Issues

How to Migrate from Python 3.6 to Python 2.7 or multiple use for both version

need to fix errors to run ensemble module with different parameters

Change plotting to take NC variable name not number

I think it makes more sense to use the variable name instead of the variable number when plotting a variable in the model outputs using the Plotting module.

(e.g., https://github.com/UW-Hydro/pysumma/blob/develop/pysumma/Plotting.py#L19)

Register SUMMA TestCases in Hydroshare

Creating a new model instance prints unwanted output

Creating a new simulation object as follows:

# Define location of .exe and file manager
executable = '/usr/local/bin/summa.exe'
file_manager = '/home/jovyan/data/pbhm-exercise-2/settings/mizoguchi/summa_fileManager_mizoguchi.txt'

# Create a model instance
s = ps.Simulation(executable, file_manager)

Generates the following output printed to the screen:

/home/jovyan/data/pbhm-exercise-2/settings/mizoguchi/ ../GENPARM.TBL

Contents of the file manager file:

controlVersion                       'SUMMA_FILE_MANAGER_V3.0.0'
simStartTime                         '2000-01-01 00:01'
simEndTime                           '2000-01-03 12:00'
tmZoneInfo                           'localTime'
settingsPath                         '/home/jovyan/data/pbhm-exercise-2/settings/mizoguchi/'
forcingPath                          '/home/jovyan/data/pbhm-exercise-2/data/mizoguchi/'
outputPath                           '/home/jovyan/data/pbhm-exercise-2/output/mizoguchi/'
decisionsFile                        'summa_zDecisions_mizoguchi.txt'
outputControlFile                    '../Model_Output.txt'
globalHruParamFile                   'summa_zLocalParamInfo.txt'
globalGruParamFile                   'summa_zBasinParamInfo.txt'
attributeFile                        'summa_zLocalAttributes.nc'
trialParamFile                       'summa_zParamTrial_mizoguchi.nc'
forcingListFile                      'summa_zForcingFileList.txt'
initConditionFile                    'summa_zInitialCond_mizoguchi.nc'
outFilePrefix                        'mizoguchi1990'
vegTableFile                         '../VEGPARM.TBL'
soilTableFile                        '../SOILPARM.TBL'
generalTableFile                     '../GENPARM.TBL'
noahmpTableFile                      '../MPTABLE.TBL'

This issue occurs on Hydroshare, unsure about other environments.

Add ability to create initial condition file

Provide more sophisticated logic for dataset merging

Currently, if there are hru and gru dimensioned variables in the output you can't simply xr.concat them because it is only possible to specify a single dimension to concatenate along. We can improve this by handling the gru and hru dimensioned variables separately, then merging that all together after.

Create GUI for Decision text file

Add ability to create input file as netcdf format

Add ability to easily perturb forcings

It would be useful to be able to perform sensitivity analysis on forcing data. Currently this is not implemented, but could be added with some helper methods on the Ensemble class or ensemble module. To accomplish this we would need to:

1. Standardize the way that forcings are perturbed - I propose we should enforce use of xarray here for predictability.
2. Standardize naming of perturbations.
3. Update the force file list to include the standardized naming of forcing perturbations.
4. Update the file manager to point to any changes to the forcing file list name.
5. Ensure interop with other types of ensembles. It should be possible to create larger sets of ensembles with forcing perturbations, parameter perturbations, and decision sets without going deep into the internals of the Ensemble configuration.

I have been running some of these types of runs in an ad hoc manner by writing multiple file managers and forcing file lists with some shell scripts then using the following lines to set up the ensembles:

dT_list = ['-4.0K', '+0.0K', '+4.0K']
config_perturbed = {
    '{}K_perturb'.format(dT): 
        {'file_manager': './file_manager_{}K.txt'.format(dT),
         'decisions': {'snowLayers': 'jrdn1991'},
         'parameters': {'windReductionParam': 0.90} 
    for dT in dT_list}

executable = 'summa.exe'
ens_dT = ps.Ensemble(executable, config_perturbed, num_workers=len(config_perturbed))

Improve scheduling for ensembles

post notebook and simulation results back to HS

Instantiation of multiple ensembles share simulations

If I create 2 ensembles via:

import pysumma as ps

exe = 'summa.exe'
fman = 'file_manager.txt'
config1 = {...some_config...}
config2 = {...some_config...}
ens1 = ps.Ensemble(exe, config1, fman)
ens2 = ps.Ensemble(exe, config2, fman)

Then ens2.simulations will contain simulations from both config1 and config2. Clearly this was not the intended behavior.

The contents of Model_Output.txt disappeared after simulation.

Add ability to create local attribute using GIS data

command line output from running summa in python

when summa is run through pysumma, can the command line output be displayed as python output? This information would be helpful especially when there are problems.

add scikit learn to dependencies

Migrate plotting functionality

I have the summa_plot repository that I started before pysumma was made. The functionality that I've implemented there should be merged into this project.

setup.py should include dependencies

It seems pysumma depends on matplotlib, xarray, and seaborn. These libraries should be listed in the install_requires.

cc @bartnijssen

Improve setup configuration to install GRASS GIS

add run_option "CUAHSI_JH"

add specwork simulation in ensemble module

Run summa with Docker using pysumma

The user should be able to run SUMMA using the SUMMA docker image via pysumma. This will likely go in execute method in the Simulation class.

more informative error msg from Simulation.execute() for incorrect run method

It would be good to have a more informative error statement in the execute method of the Simulation class if an incorrect run_method argument is passed.

Encode status strings in an enumeration type

Right now the status of the Ensemble and Simulation classes have status attributes which are explicitly referenced. It would be nice to pull this out and encode it separately into something like a Status class in a constants.py type module.

Create parent Option class

Much of the functionality in the DecisionOption class methods is also needed for the SimulationOption class. It would be nicer to have an abstract Option class that the two more specific classes can inherit from.

Unable to use BigBuckt option for groundwatr decision

When I try to change the summa groundwater decision to bigBuckt, I encounter an error which suggests that pysumma doesn't recognize this option. Notice the last line of the error message, which lists out valid options that has noXplict listed twice, and bigBuckt not listed.

Running this code:

def run_summa_error(site):
s = pysumma.Simulation.Simulation(fman_dict[site])
s.decision_obj.groundwatr.value = 'bigBuckt'
s.executable = '/opt/local/bin/summa.exe'
s.library_path = '/opt/local/lib:/opt/local/lib64:$LD_LIBRARY_PATH'
additional_args = ['export OMP_NUM_THREADS=8']
return s.execute('test_simpleResistance', 'local', additional_args)

I get:

ValueError Traceback (most recent call last)
in ()
----> 1 run_summa_error('Amplero')

in run_summa_error(site)
1 def run_summa_error(site):
2 s = pysumma.Simulation.Simulation(fman_dict[site])
----> 3 s.decision_obj.groundwatr.value = 'bigBuckt'
4 s.executable = '/opt/local/bin/summa.exe'
5 s.library_path = '/opt/local/lib:/opt/local/lib64:$LD_LIBRARY_PATH'

/pool0/data/tushark/pysumma/pysumma/Decisions.py in value(self, new_value)
77 self.write_value(self.value, new_value)
78 else:
---> 79 raise ValueError('Your input value {} is not one of the valid options {}'.format(new_value, self.options))
80
81

ValueError: Your input value bigBuckt is not one of the valid options ['qTopmodl', 'noXplict', 'noXplict']

Sensitivity Analysis

Edit Model Output

Implement shuffled complex evolution for simple calibration

Add ability to use mizuRoute for surface and river routing

remove cartopy from depencies - make optional

Add read/write functionality for "Model_Output" file

Users should be able to read and write the variable terms in the Model_Output file.

How to handle more than one netcdf output?

• Self.results_files_list = []
• When running SUMMA more than a year, it creates NetCDF file for each year.

Implement data preprocessing

Create additional hydrologic story

Add ability to use Dakota with pySUMMA

e.start() problem to execute ensemble simulation

Even though e.start() command finished in Jupyter Notebook, the output of summa is continuously generating. So we have to wait for this process finishing. Without waiting, when we try to combine output dataset from NetCDF, that makes errors.

Calibration Analysis

Create test for ensemble docker runs

apply avoidable code for methods of decision file that should not overlap

1. The LAI method option ["specified"] is no longer supported.
2. Groundwater parameterization must be (11)bigBucket when using (28)singleBasin for spatial_gw.
3. pow_prof(power-law transmissivity profile) must be selected when using (11) qTopmodl baseflow option
4. upper boundary condition for thermodynamics must be (13)presTemp with (15) (16) presHead option for soil hydrology
5. lower boundary condition for thermodynamics must be (14) zeroFlux with (15) (16) presHead option for soil hydrology
6. upper boundary condition for soil hydrology must be (15) presHead with (13)(14) presTemp and (14) zeroFlux options for thermodynamics
7. lower boundary condition for soil hydrology must be (16) zeroFlux with (11) qTopmodl option for groundwater.

Write unit tests for fileManager class

Add ability to download NLDAS forcing data

Update output control object for new style

Right now the output control object reads/writes only the old style of:

! varName          | outFreq | inst | sum | mean | var | min | max | mode
scalarSenHeatTotal | 24      | 0    | 1   | 1    | 0   | 1   | 1   | 0

We should also support the new style:

scalarSWE                     | timestep   | instant
scalarRainPlusMelt            | day        | mean

Allow blank lines in configuration files

Test “docker pull bartnijssen/summa:develop” docker version

Implement progress meter for ensembles

Creating input file as netCDF

Step-1

• Creating 1 HRU Input (LocalAttributes, InitialCond, zParamTrial) using CSV file

Step-2

• Creating 2 or more than HRUs Input (LocalAttributes, InitialCond, zParamTrial) using CSV file

Step-3

• Creating forcing input file from Website(EarthData-NLDAS 1hour, 0.125°*0.125°)

Step-4

• Creating 1 HRU Input (LocalAttributes, InitialCond, zParamTrial) using GIS data

Step-5

• Creating 2 or more than HRUs Input (LocalAttributes, InitialCond, zParamTrial) using GIS data

Improve support for distributed runs

Don't overwrite configurations with `Simulation._write_configuration`

This appears that it will solve #67. It will also simplify the running of multiple simulations and/or ensembles.