GithubHelp home page GithubHelp logo

salishseacast / tools Goto Github PK

View Code? Open in Web Editor NEW
2.0 3.0 4.0 521.01 MB

Collection of tools for working with the SalishSeaCast MEOPAR NEMO model, its results, and associated data.

Home Page: https://salishsea-meopar-tools.readthedocs.io/en/latest/index.html

License: Apache License 2.0

Jupyter Notebook 98.18% Python 1.73% Shell 0.01% MATLAB 0.01% Fortran 0.09% Mathematica 0.01%
oceanography ocean-modelling ubc-moad salishsea salishseacast

tools's Introduction

Salish Sea MEOPAR Tools

License:Apache License, Version 2.0

This is a collection of tools for working with the Salish Sea MEOPAR NEMO model, its results, and associated data.

Documentation for the tools is included in the docs/ directory and is rendered at https://salishsea-meopar-tools.readthedocs.io/en/latest/.

Documentation Status

Licenses

Unless otherwise specified, the Salish Sea MEOPAR tools and documentation are copyright 2013-2021 by the Salish Sea MEOPAR Project Contributors and The University of British Columbia.

They are licensed under the Apache License, Version 2.0. https://www.apache.org/licenses/LICENSE-2.0 Please see the LICENSE file for details of the license.

The salishsea_tools.namelist and salishsea_tools.tests.test_namelist modules are based on https://gist.github.com/krischer/4943658. They are copyright 2013 by Lion Krischer <[email protected]> and are licensed under the GNU Lesser General Public License, Version 3 (http://www.gnu.org/copyleft/lesser.html).

tools's People

Contributors

alineisabelle avatar anakarinarm avatar bmooremaley avatar brogalla avatar circularpenguin avatar dependabot[bot] avatar douglatornell avatar e-olson avatar gsgarbi avatar huangale avatar imachuca24 avatar jieliuheart avatar mbdunn avatar mdunphy avatar nsoontie avatar raif-fl avatar siddharthaanand avatar ssahu-oceans avatar susaneallen avatar tjarnikova avatar yingkaisha avatar

Stargazers

avatar avatar

Watchers

avatar avatar avatar

Forkers

mdunphy sgharbi1 cmazoyer scottveirs

tools's Issues

AttributeError: module 'numpy' has no attribute 'asscalar'

After updating numpy in nowcast-env on skookum from 1.22.3 to 1.24.0:

2022-12-21 12:26:38,380 CRITICAL [make_feeds] unhandled exception:
Traceback (most recent call last):
  File "/SalishSeaCast/NEMO_Nowcast/nemo_nowcast/worker.py", line 391, in _do_work
    checklist = self.worker_func(
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_feeds.py", line 90, in make_feeds
    max_ssh_info = _calc_max_ssh_risk(feed, run_date, run_type, config)
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_feeds.py", line 210, in _calc_max_ssh_risk
    max_ssh, max_ssh_time = _calc_max_ssh(feed, ttide, run_date, run_type, config)
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_feeds.py", line 227, in _calc_max_ssh
    ssh_ts = nc_tools.ssh_timeseries_at_point(grid_T_15m, 0, 0, datetimes=True)
  File "/SalishSeaCast/tools/SalishSeaTools/salishsea_tools/nc_tools.py", line 357, in ssh_timeseries_at_point
    time = timestamp(grid_T, range(len(ssh)), time_var=time_var)
  File "/SalishSeaCast/tools/SalishSeaTools/salishsea_tools/nc_tools.py", line 270, in timestamp
    results.append(time_orig + timedelta(seconds=np.asscalar(time_counter[i])))
  File "/SalishSeaCast/nowcast-env/lib/python3.10/site-packages/numpy/__init__.py", line 284, in __getattr__
    raise AttributeError("module {!r} has no attribute "
AttributeError: module 'numpy' has no attribute 'asscalar'

Enable `salishsea gather` to combine grid zoom output files

Original report by Doug Latornell (Bitbucket: douglatornell, GitHub: douglatornell).

The NEMO output processor appears to be incapable of putting the results from more than one grid zoom in a file. So, for example, the results for our storm surge analysis locations go into (at present) 4 separate files. Likewise, the thalweg results presently come out in 6 files.

salishsea gather should be able to do subprocess calls to tools in the NCO collection to combine those files sensibly.

Some means of specifying name patterns for files to be grouped by this mechanism should be added to the run configuration file.

Add new salishsea_tools.geo_tools.find_closest_model_point() function.

Original report by Doug Latornell (Bitbucket: douglatornell, GitHub: douglatornell).

Use code and ideas from

#!python
nowcast.figures.figures.find_model_point(
    lon, lat, X, Y, tol_lon=0.016, tol_lat=0.011)

that returns i, j, and

#!python

salishsea_tools.tide_fools.find_closest_model_point(
    lon, lat, X, Y, bathy, 
    lon_tol=0.0052, lat_tol=0.00189, allow_land=False)

that returns j, i,to create a new

#!python

salishsea_tools.geo_tools.find_closest_model_point(
    lon, lat, model_lons, model_lats,
    grid='NEMO', land_mask=None,
    tols={
        'NEMO': {tol_lon=0.0052, tol_lat=0.00189},
        'GEM2.5': {tol_lon=0.016, tol_lat=0.011},
})

that returns j, i.

model_lons, model_lats, and land_mask are all 2D arrays
(assumed to be of the same shape).
model_lons and model_lats contain floats,
and land_mask contains Booleans.
The intent is that those 3 arrays are provided by slices from netCDF4.Dataset or xarray.Dataset objects.

research_VENUS.py: load_VENUS()

Original report by Nancy Soontiens (Bitbucket: nsoontiens, ).

I use this function in a lot of places. Maybe we can move it to data_tools?

Sorry, I know we were just focusing on figures and shared but I was looking for nowcast package dependencies i my work and thought I would mention this.

Change default branch name to main

  • Change default branch name from master to main
  • Add default branch protection rule to main branch
  • Change default branch name in readthedocs Advanced Settings
  • Update clones to use main as default branch name (commands below):
git branch -m master main
git fetch origin
git branch -u origin/main main
git remote set-head origin -a

Add function to `viz_tools` to plot average wind direction arrow(s)

Original report by Doug Latornell (Bitbucket: douglatornell, GitHub: douglatornell).

It would be nice to have a function in the viz_tools module that would overlay 1 or more average wind direction arrows on surface or depth slice plots. For multiple arrows the plot region would be subdivided into equal regions; e.g. quarters. To do so the function probably needs to accept rows and cols arguments to describe how it is to be subdivided for the arrows. The arrows should be calculable from the u and v wind stress fields in the *grid_U.nc and *grid_V.nc files.

Download Weather Failing: Connection Error

Original report by Susan Allen (Bitbucket: sea_susanallen, ).

#!python

2014-12-24 09:00:24 ERROR [download_weather] Traceback (most recent call last):
2014-12-24 09:00:24 ERROR [download_weather]   File "/data/dlatorne/MEOPAR/tools/SalishSeaTools/salishsea_tools/nowcast/workers/download_weather.py", line 78, in main
2014-12-24 09:00:24 ERROR [download_weather]     get_grib(parsed_args.forecast, config, checklist)
2014-12-24 09:00:24 ERROR [download_weather]   File "/data/dlatorne/MEOPAR/tools/SalishSeaTools/salishsea_tools/nowcast/workers/download_weather.py", line 141, in get_grib
2014-12-24 09:00:24 ERROR [download_weather]     fileURL, logger, filename, retry_time_limit=9000)
2014-12-24 09:00:24 ERROR [download_weather]   File "/data/dlatorne/MEOPAR/tools/SalishSeaTools/salishsea_tools/nowcast/lib.py", line 399, in get_web_data
2014-12-24 09:00:24 ERROR [download_weather]     response = requests.get(url, stream=filepath is not None)
2014-12-24 09:00:24 ERROR [download_weather]   File "/home/dlatorne/anaconda/envs/nowcast/lib/python2.7/site-packages/requests/api.py", line 60, in get
2014-12-24 09:00:24 ERROR [download_weather]     return request('get', url, **kwargs)
2014-12-24 09:00:24 ERROR [download_weather]   File "/home/dlatorne/anaconda/envs/nowcast/lib/python2.7/site-packages/requests/api.py", line 49, in request
2014-12-24 09:00:24 ERROR [download_weather]     return session.request(method=method, url=url, **kwargs)
2014-12-24 09:00:24 ERROR [download_weather]   File "/home/dlatorne/anaconda/envs/nowcast/lib/python2.7/site-packages/requests/sessions.py", line 457, in request
2014-12-24 09:00:24 ERROR [download_weather]     resp = self.send(prep, **send_kwargs)
2014-12-24 09:00:24 ERROR [download_weather]   File "/home/dlatorne/anaconda/envs/nowcast/lib/python2.7/site-packages/requests/sessions.py", line 569, in send
2014-12-24 09:00:24 ERROR [download_weather]     r = adapter.send(request, **kwargs)
2014-12-24 09:00:24 ERROR [download_weather]   File "/home/dlatorne/anaconda/envs/nowcast/lib/python2.7/site-packages/requests/adapters.py", line 407, in send
2014-12-24 09:00:24 ERROR [download_weather]     raise ConnectionError(err, request=request)
2014-12-24 09:00:24 ERROR [download_weather] ConnectionError: ('Connection aborted.', gaierror(-2, 'Name or service not known'))

Add time_origin attritbut to netCDF files produced by grib_to_netcdf worker

Original report by Doug Latornell (Bitbucket: douglatornell, GitHub: douglatornell).

The time_counter variable in the netCDF files that the grib_to_netcdf worker produces should have a time_origin attribute that is set to 1970-01-01 00:00:00.

The lack of that attribute results in code like https://bitbucket.org/salishsea/tools/src/55a6bed6f24d12f82884118b1bd8ce1a51900586/SalishSeaNowcast/nowcast/figures.py?at=default&fileviewer=file-view-default#figures.py-722

When this issue has been resolved the code in nc_tools.uv_wind_timeseries()
and tests.test_nc_tools.test_uv_wind_timeseries_time_counter_type() needs to be updated to make use of time_origin.

Once the time_origin attribute is being included in new ops*.nc files we need a maintenance script to run over the previously created files to add the attribute.

shared.py: _make_backgound_map() and plot_map()

Original report by Nancy Soontiens (Bitbucket: nsoontiens, ).

These functions could be really useful outside of nowcast, especially when trying to produce figures for papers or talks. Maybe in geo_tools?

This isn't super urgent, I don't think I actually call this anywhere but it was a thought as a read through the module.

Make nowcast deployment on salish runnable by anyone

Original report by Doug Latornell (Bitbucket: douglatornell, GitHub: douglatornell).

There are lots of file ownership issues that have the effect limiting much of the nowcast deployment on salish to being run only by dlatorne.

One solution that would allow other authorized users to run the system would be to deploy it under a system-specific user (e.g nowcast) so that operations could be performed after doing su nowcast. However, in the absence of sudo privileges for the authorized users group, a shared password would have to be used.

Another possible solution would be to run the nowcast system in a collection of Docker containers.

DatasetID needs to include run type

Original report by Doug Latornell (Bitbucket: douglatornell, GitHub: douglatornell).

The initial test Point Atkinson sea surface height dataset id is presently ubcSSNPointAtkinsonSSH15m, but that makes no accounting for run type
(nowcast, forecast, forecast2, nowcast-green).
Possible alternatives:

  • ubcSSNNowcastPointAtkinsonSSH15m, but that implies ubcSSNNowcast-greenPointAtkinsonSSH15m, which is rather long
  • ubcSSNNowPointAtkinsonSSH15m, which implies ubcSSNForePointAtkinsonSSH15m, ubcSSNFore2PointAtkinsonSSH15m, ubcSSNNow-greenPointAtkinsonSSH15m, but those are somewhat (even more) cryptic

It is also possible that the whole "formula" for the dataset ids needs to change.

Make NEMO-code path handling more robust

Original report by Doug Latornell (Bitbucket: douglatornell, GitHub: douglatornell).

The relative path to the NEMO-code directory may differ between the run-sets files directory and the temporary run directory, so a path that works for salishsea prepare may fail for salishsea combine.

This issue was noted during testin of the idea of putting individual team members run-set files in their own directories under SS-run-sets/SalishSea/.

A possible resolution would be to calculate the path to NEMO-code from the nemo.exe symlink in the run directory.

Add email listener to detect end of nowcast runs on orcinus

Original report by Doug Latornell (Bitbucket: douglatornell, GitHub: douglatornell).

If a new worker that listened for email messages from the queue manager on orcinus were launched when the run was queued there we should be able to detect the start and end of the run. The start is just log information, but the detection of the end of the run should allow downloading of the results and the rest of the automation to proceed without human intervention.

For orcinus runs to be fully automated we need a way to prepare the namelist.time and nowcast.yaml files and queue the run on orcinus after the make_forcing_links worker finishes its job.

Move results of `salishsea combine` to a directory

Original report by Doug Latornell (Bitbucket: douglatornell, GitHub: douglatornell).

Add functionality to move the results of salishsea combine to a specified directory. The interface will change to add the results directory as a required command line argument:

#!bash
salishsea RUN_DESC RESULTS_DIR

The present functionality of leaving the combined results files in the current directory would be obtained with:

#!bash
salishsea RUN_DESC ./

Make research_VENUS.load_VENUS() fault tolerant

Original report by Doug Latornell (Bitbucket: douglatornell, GitHub: douglatornell).

load_VENUS() uses request.get() without any re-try log. It almost always works (and is fine for notebook use), but when a ConnectionError occurs in the nowcast system's use case it stops the automation path to pushing the results to the web (happene on 11dec15).

Idea:

  • Split load_VENUS() to separate getting the data from the ONC web service and parsing it into a pandas dataframe.
  • Retain a load_VENUS() function that gets data from the web for use in notebooks.
  • Change compare_VENUS() to accept either a buffer containing the web data, or a pandas dataframe as an arg.
  • Do the data download using nowcast.lib.get_web_data() in make_plots._make_research_plots() and pass the data into compare_VENUS().

Need to think about/explore binary vs. text data handling as nowcast.lib.get_web_data() presently only handles binary (response.content).

Retrieve stdout & stderr files for TORQUE runs on salish

Original report by Doug Latornell (Bitbucket: douglatornell, GitHub: douglatornell).

Runs submitted via qsub on salish produce stdout and stderr files that end up in /var/spool/torque/undelivered/ with names like job_id.OU and job_id.ER; e.g. 75.master.OU and 75.master.ER. They should be moved to the run results directory as a result of the #PBS -o and #PBS -e directives in SalishSeaNEMO.sh, but they are not.

Once the job is queued its job_id can be obtained with the shell pipe:

#!bash

qstat | grep job_name | cut -d" " -f1

So, the salishsea run command needs to be modified so that when it is run on salish code is added to SalishSeaNEMO.sh to get the job_id and use it to move the stdout and stderr files from /var/spool/torque/undelivered/ to the run results directory at the very end of processing.

Set permissions so any wg-moad user can do nowcast runs on orcinus

Original report by Doug Latornell (Bitbucket: douglatornell, GitHub: douglatornell).

  • upload_forcing, upload_all_files, and make_forcing_links workers need to set group and permissions appropriately
  • /home/dlatorne/MEOPAR/nowcast/namelist.time and /home/dlatorne/MEOPAR/nowcast/nowcast.yaml need to the group writable
  • appropriate salishsea run command needs to be documented; e.g. salishsea run nowcast.yaml iodef.xml ../SalishSea/nowcast/15jul15
  • docs for the process need to be written

Download Weather Failing: Error 104

Original report by Susan Allen (Bitbucket: sea_susanallen, ).

#!python

2014-12-21 22:33:42 ERROR [download_weather] Traceback (most recent call last):
2014-12-21 22:33:42 ERROR [download_weather]   File "/data/dlatorne/MEOPAR/tools/SalishSeaTools/salishsea_tools/nowcast/workers/download_weather.py", line 78, in main
2014-12-21 22:33:42 ERROR [download_weather]     get_grib(parsed_args.forecast, config, checklist)
2014-12-21 22:33:42 ERROR [download_weather]   File "/data/dlatorne/MEOPAR/tools/SalishSeaTools/salishsea_tools/nowcast/workers/download_weather.py", line 141, in get_grib
2014-12-21 22:33:42 ERROR [download_weather]     fileURL, logger, filename, retry_time_limit=9000)
2014-12-21 22:33:42 ERROR [download_weather]   File "/data/dlatorne/MEOPAR/tools/SalishSeaTools/salishsea_tools/nowcast/lib.py", line 402, in get_web_data
2014-12-21 22:33:42 ERROR [download_weather]     return _handle_url_content(response, filepath)
2014-12-21 22:33:42 ERROR [download_weather]   File "/data/dlatorne/MEOPAR/tools/SalishSeaTools/salishsea_tools/nowcast/lib.py", line 444, in _handle_url_content
2014-12-21 22:33:42 ERROR [download_weather]     for block in response.iter_content(1024):
2014-12-21 22:33:42 ERROR [download_weather]   File "/home/dlatorne/anaconda/envs/nowcast/lib/python2.7/site-packages/requests/models.py", line 653, in generate
2014-12-21 22:33:42 ERROR [download_weather]     for chunk in self.raw.stream(chunk_size, decode_content=True):
2014-12-21 22:33:42 ERROR [download_weather]   File "/home/dlatorne/anaconda/envs/nowcast/lib/python2.7/site-packages/requests/packages/urllib3/response.py", line 256, in stream
2014-12-21 22:33:42 ERROR [download_weather]     data = self.read(amt=amt, decode_content=decode_content)
2014-12-21 22:33:42 ERROR [download_weather]   File "/home/dlatorne/anaconda/envs/nowcast/lib/python2.7/site-packages/requests/packages/urllib3/response.py", line 186, in read
2014-12-21 22:33:42 ERROR [download_weather]     data = self._fp.read(amt)
2014-12-21 22:33:42 ERROR [download_weather]   File "/home/dlatorne/anaconda/envs/nowcast/lib/python2.7/httplib.py", line 567, in read
2014-12-21 22:33:42 ERROR [download_weather]     s = self.fp.read(amt)
2014-12-21 22:33:42 ERROR [download_weather]   File "/home/dlatorne/anaconda/envs/nowcast/lib/python2.7/socket.py", line 380, in read
2014-12-21 22:33:42 ERROR [download_weather]     data = self._sock.recv(left)
2014-12-21 22:33:42 ERROR [download_weather] error: [Errno 104] Connection reset by peer

figures.py: ssh data functions

Original report by Nancy Soontiens (Bitbucket: nsoontiens, ).

I have some notebooks that I periodically run through to compare nowcast ssh with observations. They make use of functions:

load_archived_observations()
dateparse_archive_obs()
get_NOAA_wlevels()
get_NOAA_tides()
dateparse_NOAA()

I can see these being useful outside of the nowcast package or nowcast comparisons. Even just to get observed water levels over a time period.

Perhaps they can move to data_tools?

Special handling for NEMO restart files in `salishsea combine`

Original report by Doug Latornell (Bitbucket: douglatornell, GitHub: douglatornell).

NEMO restart file will generally not be downloaded,
need to be uncompressed for use as the initial state for a subsequent run,
and are sometimes not needed at all.

  • Change the default behaviour to not compress restart files
  • Add a --compress-restart option to override the default behaviour
  • Add a --delete-restart option to delete the pre-processor restart files instead of combining and compressing them

Handle missing Neah Bay ssh observations files

Original report by Doug Latornell (Bitbucket: douglatornell, GitHub: douglatornell).

The NOAA Neah Bay site occasionally fails to provide an sea surface height observations file, and that causes nowcast to stumble. We should be able to avoid the stumble by symlinking an appropriate ssh forecast file as the observations file in the upload_forcing worker (and probably the upload_all_files worker too, for completeness).

AttributeError: module 'numpy' has no attribute 'float'

After updating numpy in nowcast-env environment from 1.22.3 to 1.24.0:

2022-12-22 07:26:14,218 CRITICAL [make_live_ocean_files] unhandled exception:
Traceback (most recent call last):
  File "/SalishSeaCast/NEMO_Nowcast/nemo_nowcast/worker.py", line 391, in _do_work
    checklist = self.worker_func(
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_live_ocean_files.py", line 82, in make_live_ocean_files
    filepaths = create_LiveOcean_TS_BCs(
  File "/SalishSeaCast/tools/SalishSeaTools/salishsea_tools/LiveOcean_BCs.py", line 605, in create_LiveOcean_TS_BCs
    interpl = stabilize(sigmal, interpl)
  File "/SalishSeaCast/tools/SalishSeaTools/salishsea_tools/LiveOcean_BCs.py", line 238, in stabilize
    interps['salt'][:k+1, i, j] += -add_salt/np.float(k+1)
  File "/SalishSeaCast/nowcast-env/lib/python3.10/site-packages/numpy/__init__.py", line 284, in __getattr__
    raise AttributeError("module {!r} has no attribute "
AttributeError: module 'numpy' has no attribute 'float'

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.