csyhuang / hn2016_falwa

Python library for computing Finite-Amplitude Local Wave Activity from climate data.

Python Library: falwa (v1.2.1)

Important: this python package has been renamed from hn2016_falwa to falwa since version v1.0.0.

Compute from gridded climate data the Finite-amplitude Local Wave Activity (FALWA) and flux terms introduced in:

• Huang and Nakamura (2016, JAS)
• Huang and Nakamura (2017, GRL).
• Nakamura and Huang (2018, Science) Atmospheric Blocking as a Traffic Jam in the Jet Stream.
• Neal et al (2022, GRL) The 2021 Pacific Northwest Heat Wave and Associated Blocking: Meteorology and the Role of an Upstream Cyclone as a Diabatic Source of Wave Activity

Package Installation

This current version works for Python 3.x. Note that since v0.3.0, some functions have backend in Fortran. To build the package from source, you need a fortran compiler (e.g. gfortran) to implement the installation.

Since the package is still being actively developed, please use the develop mode for installation.

To install the package for the first time, clone the GitHub repo and install via develop mode:

git clone https://github.com/csyhuang/hn2016_falwa.git
cd hn2016_falwa
python setup.py develop

To incorporate updates, pull the new version of the code from GitHub. Remove any existing f2py modules and recompile.

# Assume you are already in the hn2016_falwa/ repo
git pull
rm falwa/*.so
python setup.py develop
pytest tests/ # to check if the package can be run properly

Quick start

There are some readily run python scripts (in scripts/) and jupyter notebooks (in notebooks/) which you can start with. The netCDF files needed can be found in Clare's Dropbox folder.

Depending on what you want to do, the methods to be use may be different.

1. If you solely want to compute equivalent latitude and local wave activity from a 2D field, you can refer to notebooks/simple/Example_barotropic.ipynb. This is useful for users who want to use LWA to quantify field anomalies.

2. If you want to compute zonal wind reference states and wave activity fluxes in QG Formalism, look at notebooks/nh2018_science/demo_script_for_nh2018.ipynb for the usage of QGField. This notebook demonstrates how to compute wave activity and reference states presented in Nakamura and Huang (2018). To make sure the package is properly installed in your environment, run through the notebook after installation to see if there is error.

THe conda environment for running the notebook can be found in environment.yml. To create the conda environment, execute:

conda env create -f environment.yml

Sponsorship acknowledgement

This project is sponsored by JetBrains as one of the non-commercial open source projects. JetBrains provides core project contributors with a set of best-in-class developer tools free of charge. Please check out their Open Source Support page for details.

Inquiries / Issues reporting

• If you are interested in using the package, please leave your contact here or email me([email protected]) such that I can keep you updated of any changes made.
• If you encounter coding issues/bugs when using the package, please create an Issue ticket.
• If you have scientific questions, please contact Clare S. Y. Huang via email([email protected]).

hn2016_falwa's Issues

To compute non-conservative forcing with lwa

include that in lwa

Research into Scikit-build solution for F2PY

Explore whether scikit-build would be a better solution for compiling F2PY modules:
https://scikit-build.readthedocs.io/en/latest/cmake-modules/F2PY.html

An example of project using scikit-build
https://github.com/fastscape-lem/fastscapelib-fortran/blob/master/pyproject.toml

FYI, NSF grant associated with this project:
https://www.nsf.gov/awardsearch/showAward?AWD_ID=2209877

Errors: Calculations in the southern hemisphere

Hi Clare,

I used GFDL output to run the program and it worked well with only the northern hemisphere. However, there is an error when I turn "northern_hemisphere_results_only" into false. Here I post the error message:

---

ValueError Traceback (most recent call last)
in ()
18 qgpv[tstep, :, :, :], interpolated_u[tstep, :, :, :], interpolated_v[tstep, :, :, :], interpolated_theta[tstep, :, :, :], static_stability = qgfield_object.interpolate_fields()
19
---> 20 qref[tstep, :, :], uref[tstep, :, :], ptref[tstep, :, :] = qgfield_object.compute_reference_states(northern_hemisphere_results_only=False)
21
22 adv_flux_f1[tstep, :, :], adv_flux_f2[tstep, :, :], adv_flux_f3[tstep, :, :], adv_flux_conv[tstep, :, :], divergence_eddy_momentum_flux[tstep, :, :], meridional_heat_flux[tstep, :, :], lwa_baro[tstep, :, :], u_baro[tstep, :, :], lwa[tstep, :, :, :] = qgfield_object.compute_lwa_and_barotropic_fluxes(northern_hemisphere_results_only=False)

/home/hungi/Documents/Localized_Wave/hn2016_falwa/hn2016_falwa/oopinterface.pyc in compute_reference_states(self, northern_hemisphere_results_only)
414 np.swapaxes(self._ptref_temp, 0, 1)))
415
--> 416 return self.qref, self.uref, self.ptref
417
418 @Property

/home/hungi/Documents/Localized_Wave/hn2016_falwa/hn2016_falwa/oopinterface.pyc in qref(self)
423 if self._qref is None:
424 raise ValueError('qref is not computed yet.')
--> 425 return self._return_interp_variables(variable=self._qref, interp_axis=1)
426
427 @Property

/home/hungi/Documents/Localized_Wave/hn2016_falwa/hn2016_falwa/oopinterface.pyc in _return_interp_variables(self, variable, interp_axis, northern_hemisphere_results_only)
299 variable, self.ylat[-(self.nlat//2+1):],
300 self.ylat_no_equator[-(self.nlat//2):],
--> 301 which_axis=interp_axis)
302 else:
303 return self._interp_back(variable, self.ylat, self.ylat_no_equator, which_axis=interp_axis)

/home/hungi/Documents/Localized_Wave/hn2016_falwa/hn2016_falwa/oopinterface.pyc in _interp_back(self, field, interp_from, interp_to, which_axis)
226 return interp1d(
227 interp_from, field, axis=which_axis, bounds_error=False,
--> 228 fill_value='extrapolate'
229 )(interp_to)
230

/home/hungi/anaconda2/lib/python2.7/site-packages/scipy/interpolate/interpolate.pyc in init(self, x, y, kind, axis, copy, bounds_error, fill_value, assume_sorted)
431 assume_sorted=False):
432 """ Initialize a 1D linear interpolation class."""
--> 433 _Interpolator1D.init(self, x, y, axis=axis)
434
435 self.bounds_error = bounds_error # used by fill_value setter

/home/hungi/anaconda2/lib/python2.7/site-packages/scipy/interpolate/polyint.pyc in init(self, xi, yi, axis)
58 self.dtype = None
59 if yi is not None:
---> 60 self._set_yi(yi, xi=xi, axis=axis)
61
62 def call(self, x):

/home/hungi/anaconda2/lib/python2.7/site-packages/scipy/interpolate/polyint.pyc in _set_yi(self, yi, xi, axis)
123 shape = (1,)
124 if xi is not None and shape[axis] != len(xi):
--> 125 raise ValueError("x and y arrays must be equal in length along "
126 "interpolation axis.")
127

ValueError: x and y arrays must be equal in length along interpolation axis.

Conda environment outdated

There is an error when setting up conda environment using hn2016_falwa/examples/environment.yml. Thanks Martin Pecnjak for reporting.

ResolvePackageNotFound:
  - libxml2==2.9.8=0
  - nbconvert==5.3.1=py_1
  - libuuid==1.0.3=1
  - libssh2==1.8.0=2
  - matplotlib==2.2.2=py36_1
  - jpeg==9b=2
  - libiconv==1.15=0

Local FAWA v.s. local APE

Hi, @csyhuang

I've recently notice the possible similarity between your local FAWA and local APE (available potential energy). After reading two JFM papers by Scotti and White (2014) and Winters and Barkan (2013), I found that:

1. both of them defined in a contour-based coordinate;
2. in small-amplitude limits, both of them reduce to the classical definitions;
3. locally positive definite;

Not so sure if we can prove their equivalence mathematically. But if it is, I think one could use the local FAWA in identifying local internal wave breaking events (vertically), given that it is useful in identifying the local Rossby wave breaking events (horizontally). Do you have any comments on this?

possible bug in lwa_theta function

Hi Clare,
I think there might be a bug in the 'lwa_theta' function for the recent version of hn2016_falwa.
Here is the error:

File "/home/slubis/Enthought/Canopy_64bit/User/lib/python2.7/site-packages/hn2016_falwa-0.2.0-py2.7.egg/hn2016_falwa/wrapper.py", line 551, in theta_lwa n_points, planet_radius=planet_radius) NameError: global name 'n_points' is not defined

The function is called by hn2016_falwa.wrapper import theta_lwa.
Thank you,

Incorporate listed diagnostic to MDTF and submit pull request

Submit the first pull request to MDTF repository:

• Use the short name finite_amplitude_wave_diag as branch name and diagnostic abbreviation. Full name would be Finite-amplitude Rossby wave diagnostics
• Incorporate diagnostics listed in #79
• Specify a reasonable time window in .jsonc and fieldlist such that local test can finish in reasonable amount of time
• Make sure the code runs both on Mac and Linux (currently, conda install is only available on Linux. #80 and potentially some new tickets are to solve the anaconda package deployment issues on Mac.

This ticket can be split to smaller ones depending on the capacity of the contributor.

The checklist for submitting pull request to MDTF repo:

• I have reviewed my own code to ensure that if follows the POD development guidelines
• My branch is up-to-date with the NOAA-GFDL develop branch, and all merge conflicts are resolved
• The script are written in Python 3.6 or above (preferred; required if funded by a CPO grant), NCL, or R
• All of my scripts are in the diagnostics/[POD short name] subdirectory, and include a main_driver script, template html, and settings.jsonc file
• I have made corresponding changes to the documentation in the POD's doc/ subdirectory
• If applicable, I've added a .yml file to src/conda, and my environment builds with conda_env_setup.sh
• I have added any necessary data to input_data/obs_data/[pod short name] and/or input_data/model/[pod short name]
• My code is portable; it uses MDTF environment variables, and does not contain hard-coded file or directory paths
• I have provided the code to generate digested data files from raw data files
• Each digested data file generated by the script contains numerical data (no figures), and is 3 GB or less in size
• The repository contains no extra test scripts or data files

ValueError: The reference state does not converge for Northern Hemisphere.

Hello again! I was doing a trial run with some data I had and ran into the given error. The interpolated fields look good to me but qgfield_object.compute_reference_states(northern_hemisphere_results_only=False) fails. I will take a closer look at the values tomorrow but please let me know if I should look for something specific for debugging this. Thanks in advance.

Cope with the differences between treatment of fields in QGField in NH18 and NHN22

There are a few differences between the computation in NH18 and NHN22:

NH18

1. QGPV contains smoothing term
2. A global t0 and static stabilitiy are used.
3. Qref is computed in global domain
4. Uref is computed using equator as equatorward latitude boundary

NHN22

1. QGPV does not contain smoothing term
2. Hemispheric t0 and static stability are used
3. Qref is computed in global domain
4. Uref is computed using 5^\circ as equatorward latitude boundary

It would be ideal if users can plug-and-play with different options, but given time constraint, in release0.7.0 I would try to encapsulate all these into QGField minimally such that users can reproduce results in the papers with QGField.

Issues with deployment onto conda channel

So far, deployment onto pip/conda channel has not been successful yet.

Here is what I tried for conda deployment.

My build.sh:

$PYTHON setup.py install

My meta.yaml

{% set name = "hn2016_falwa" %}
{% set version = "0.7.0" %}

package:
  name: "{{ name|lower }}"
  version: "{{ version }}"

source:
  git_url: https://github.com/csyhuang/hn2016_falwa

requirements:
  build:
    - python
    - numpy==1.22.3    
    - setuptools
  run:
    - python

test:
  imports:
    - hn2016_falwa

about:
  home: https://github.com/csyhuang/hn2016_falwa
  license: MIT
  license_family: MIT
  summary: Finite amplitude local wave activity package

I first created a conda environment test_MDTF_base from the MDTF base environment, then I did:

conda build .

and a compatibility issue between the specific numpy version used and setuptools arises:

Solving environment: ...working... 
Found conflicts! Looking for incompatible packages.
This can take several minutes.  Press CTRL-C to abort.
failed                                                                                                                                                          
failed to parse packages from exception: The following specifications were found to be incompatible with each other:

Output in format: Requested package -> Available versions

Package python conflicts for:
setuptools -> python[version='>=2.7,<2.8.0a0|>=3.10,<3.11.0a0|>=3.8,<3.9.0a0|>=3.9,<3.10.0a0|>=3.7,<3.8.0a0|>=3.6,<3.7.0a0|>=3.11,<3.12.0a0|>=3.5,<3.6.0a0']
numpy==1.22.3 -> mkl_fft[version='>=1.3.1,<2.0a0'] -> python[version='2.7.*|3.6.*|>=2.7,<2.8.0a0|>=3.6,<3.7.0a0|>=3.7,<3.8.0a0|>=3.5,<3.6.0a0']
python=3.7
numpy==1.22.3 -> python[version='>=3.10,<3.11.0a0|>=3.9,<3.10.0a0|>=3.8,<3.9.0a0|>=3.11,<3.12.0a0']
setuptools -> certifi[version='>=2016.9.26'] -> python[version='2.7.*|3.6.*']

Package tzdata conflicts for:
setuptools -> python[version='>=3.9,<3.10.0a0'] -> tzdata
numpy==1.22.3 -> python[version='>=3.10,<3.11.0a0'] -> tzdata

Verify that all example notebooks and scripts are working

All netCDF files used in example notebooks/scripts can be found in my Dropbox:
https://www.dropbox.com/scl/fo/b84pwlr7zzsndq8mpthd8/h?dl=0&rlkey=f8c1gm2xaxvx3c7cf06vop6or
Verify that all example notebooks (notebooks/) and scripts (scripts/) on branch release0.7.0 are producing expected results.

Provide default variable encoding parameters for xarray's to_netcdf

xarray.Dataset.to_netcdf has an option for encoding, allowing data to be written as int16, saving 75% of disk space compared to direct storage of double and 50% compared to float. Compression can additionally be applied for further savings.

The encoding requires an offset and scale factor to be chosen for each variable, so that the values can be stored as

value_unpacked = scale_factor * value_packed + add_offset

The (potential) loss of precision is usually no problem for atmospheric variables (most reanalysis data is delivered like this anyway), as long as the scale factor and offset are appropriately chosen. This is where some specific knowledge about sensible ranges for each field stored is required.

Users could always choose their own, but we might want to provide a default set of encodings for the output variables of the QGField properties. They need to be general enough to match any season, hemisphere, region, etc. but specific enough to retain precision. I suggest we work out a set of values in this issue and ship these conveniently with the package eventually.

See also:

• https://docs.unidata.ucar.edu/nug/current/best_practices.html#bp_Packed-Data-Values
• http://james.hiebert.name/blog/work/2015/04/18/NetCDF-Scale-Factors.html

Problem importing QGFied : from hn2016_falwa.oopinterface import QGField

Hi. Dr. Huang

This is Amin Fazl Kazemi. Sorry for sending lots of issues
I sent you an email for issue : "Problem using qgfield_object.compute_lwa_and_barotropic_fluxes()"

Hope hear the answer.

I also faced problem installing the package in windows. I nstalled Ubuntu 21 linux over vmware workstation hyper visor and problem is solved. Thanks in advance

Generalize hemisphere_to_global to be used for both numpy and xarray interface

This functionality is useful at several places, e.g. computation of LWA flux vectors, and handling hemispheric data input.

See #45 for details.

Release 0.7.0

This is a tentative list of new features/functionality in release 0.7.0. The target release date would be 2023/6/1 (it may change depending on the progress of publication in our group).

Additional functionalities to include:

• #46
• Redesign the readme.md and make it a roadmap, e.g. "What do you want to do? If (A), go to (module name)..."
• Better encapsulate the option northern_hemisphere_results_only in QGField.
• (tentative) Non-QG formalism of LWA

The following item may be left for the next release:

• Comprehensive unit tests for QGField

---

The two items below will be left for release v0.7.1 #75

• #50
• #48

Reminder for users:

• Inversion using intel's numpy and scipy is much faster

Grid points issues in LWA flux function

Hi Clare,

Thanks for the clean and useful codes/subroutines. It seems that the functions do not work properly for the even grid points, but works well for odd grid points. Could you please check it? Probably there is a minor bug somewhere but I can not find :)

Thanks, Clare!

netCDF4 installation requirement has to be updated

Right now it takes forever to include netcdf4 in the conda environment.

Computing effective diffusive flux - divided by zero

In basis.eqvlat, when computing the effective diffusive flux (brac_return), there is an operation where the surface integral of vorticity gradient is divided by the differential area:

        if vgrad is not None:
            # Prone to error when aa[i] -> 0
            dp[i] = np.sum(area_flat[np.where(inds == i)] *
                           vgrad_flat[np.where(inds == i)]) / aa[i]

There has to be an error or a corresponding operation when aa[i] is zero.

This issue came up in the unittest test_eqvlat in test_basis.py.

ValueError: masked arrays are not supported

Hi Claire,
I successfully run your example "demo_script_for_nh2018" (I copied the code from the Jupiter notebook in a .py file and I run the code on my machine). However, when I try to use the same code with my data (model output interpolated to a regular lat/lon grid) I got the following error:

Dimension of time: 124
Traceback (most recent call last):

  File "<ipython-input-1-26269db047c2>", line 1, in <module>
    runfile('/mnt/6d042d15-9e3c-4da4-b065-f073e9857fcb/Dropbox/python/LWA/hn2016_falwa/examples/nh2018_science/demo_script_for_nh2018_GUIDO.py', wdir='/mnt/6d042d15-9e3c-4da4-b065-f073e9857fcb/Dropbox/python/LWA/hn2016_falwa/examples/nh2018_science')

  File "/home/guido/anaconda3/lib/python3.7/site-packages/spyder_kernels/customize/spydercustomize.py", line 827, in runfile
    execfile(filename, namespace)

  File "/home/guido/anaconda3/lib/python3.7/site-packages/spyder_kernels/customize/spydercustomize.py", line 110, in execfile
    exec(compile(f.read(), filename, 'exec'), namespace)

  File "/mnt/6d042d15-9e3c-4da4-b065-f073e9857fcb/Dropbox/python/LWA/hn2016_falwa/examples/nh2018_science/demo_script_for_nh2018_GUIDO.py", line 159, in <module>
    interpolated_theta[tstep, :, :, :], static_stability = qgfield_object.interpolate_fields()

  File "/mnt/6d042d15-9e3c-4da4-b065-f073e9857fcb/Dropbox/python/LWA/hn2016_falwa/hn2016_falwa/oopinterface.py", line 401, in interpolate_fields
    print(isinstance(self.qgpv,np.ma.MaskedArray))

  File "/mnt/6d042d15-9e3c-4da4-b065-f073e9857fcb/Dropbox/python/LWA/hn2016_falwa/hn2016_falwa/oopinterface.py", line 695, in qgpv
    variable=self._qgpv, interp_axis=1, northern_hemisphere_results_only=False)

  File "/mnt/6d042d15-9e3c-4da4-b065-f073e9857fcb/Dropbox/python/LWA/hn2016_falwa/hn2016_falwa/oopinterface.py", line 284, in _return_interp_variables
    return self._interp_back(variable, self.ylat, self.ylat_no_equator, which_axis=interp_axis)

  File "/mnt/6d042d15-9e3c-4da4-b065-f073e9857fcb/Dropbox/python/LWA/hn2016_falwa/hn2016_falwa/oopinterface.py", line 257, in _interp_back
    )(interp_to)

  File "/home/guido/anaconda3/lib/python3.7/site-packages/scipy/interpolate/polyint.py", line 78, in __call__
    x, x_shape = self._prepare_x(x)

  File "/home/guido/anaconda3/lib/python3.7/site-packages/scipy/interpolate/polyint.py", line 90, in _prepare_x
    x = _asarray_validated(x, check_finite=False, as_inexact=True)

  File "/home/guido/anaconda3/lib/python3.7/site-packages/scipy/_lib/_util.py", line 237, in _asarray_validated
    raise ValueError('masked arrays are not supported')

ValueError: masked arrays are not supported

The problem seems to be related with the qgfield_object.qgpv variable. In fact, I got the same error when I try to directly access to this variable:

qgfield_object.qgpv

Traceback (most recent call last):

  File "<ipython-input-9-57090d46ce92>", line 1, in <module>
    qgfield_object.qgpv

  File "/mnt/6d042d15-9e3c-4da4-b065-f073e9857fcb/Dropbox/python/LWA/hn2016_falwa/hn2016_falwa/oopinterface.py", line 695, in qgpv
    variable=self._qgpv, interp_axis=1, northern_hemisphere_results_only=False)

  File "/mnt/6d042d15-9e3c-4da4-b065-f073e9857fcb/Dropbox/python/LWA/hn2016_falwa/hn2016_falwa/oopinterface.py", line 284, in _return_interp_variables
    return self._interp_back(variable, self.ylat, self.ylat_no_equator, which_axis=interp_axis)

  File "/mnt/6d042d15-9e3c-4da4-b065-f073e9857fcb/Dropbox/python/LWA/hn2016_falwa/hn2016_falwa/oopinterface.py", line 257, in _interp_back
    )(interp_to)

  File "/home/guido/anaconda3/lib/python3.7/site-packages/scipy/interpolate/polyint.py", line 78, in __call__
    x, x_shape = self._prepare_x(x)

  File "/home/guido/anaconda3/lib/python3.7/site-packages/scipy/interpolate/polyint.py", line 90, in _prepare_x
    x = _asarray_validated(x, check_finite=False, as_inexact=True)

  File "/home/guido/anaconda3/lib/python3.7/site-packages/scipy/_lib/_util.py", line 237, in _asarray_validated
    raise ValueError('masked arrays are not supported')

ValueError: masked arrays are not supported

Have you any idea on how to solve the issue? My date are apparently ok, without missing values, and I prepared them mimicking the structure of the example data.
Thank you!

Generalize the direct inversion algorithm to handle data of arbitrary lon/lat resolution

The additional direct inversion algorithm assumes 1°-resolution data (since we were in a hurry submitting Neal et al with the code). Generalize the code to handle data of arbitrary latitudinal resolution (uniform grid).

See pull request from Chris for details #45

Trying to run a notebook in examples/nh2018_science. Unable to import modules: __init__.py refers to modules not present in hn_2016_falwa

Deploy the package on Anaconda channel

1. Deploy the package hn2016_falwa on conda such that users can install the package via conda install ....
2. Document the procedures for deployment - given we are the only external contributors collaborating with MDTF team via anaconda channel, our experience is valuable for future external contributors.
3. Test that everything (conda install, package import, etc) works as expected.

Note: it seems not necessary for this to be done after the f2py-to-cython translation. This can be done any time as long as the deployment procedures are well-documented and repeatable.

Enhance QGField.compute_reference_states to compute ref states in Southern Hemisphere

Right now, the compute_reference_states method in the class QGField only deal with data in the northern hemisphere. Would be good to include functionality to compute relevant fields in the southern hemisphere.

Solve Memory Issues of QGField

A new QGField object is created at every timestamp. Figure out a more memory-efficient way to handle time-series data.

See pull request from Chris for details #45

Direct Solver to solve u_ref

This is suggestion from Noboru - to implement direct solver to solve uref from elliptic equation. The algorithm is outlined in:

Lindzen, R. S., & Kuo, H. L. (1969). A reliable method for the numerical integration of a large class of ordinary and partial differential equations. In Mon. Wea. Rev.

I would include the Lindzen and Kuo solver as an option in the QGField, such that there are two solvers (SOR v.s. direct solver) user can choose from.

Different in UREF in NS10 and NH18

(Question thread) Different in UREF in NS10 and NH18

To-do items for Release 1.0

Update: The new version (v1.0) will be released by Nov 1, 2023 tentatively. We plan to change the python package name from hn2016_falwa to falwa.

Update [2023/11/5]: For testing the MDTF integration, I will deploy a pre-release version 1.0.0a0 from the branch pre-release-1.0.0a0 with the following enhancement/fixes:

• Python package name change from hn2016_falwa to falwa #91
• QGDataset does not support data input with even number of latitude gridpoints #85
• Use Cartopy instead of Basemap (deprecated) in all the visualization examples #87
• Bug in GitHub action #82
• Code fix for zonal advective flux computation #92

Update [2023/11/5]: The rest of the issues will be addressed in upcoming release(s):

• Solve Memory Issues of QGField #50
• Use pyproject.toml for setup #73
  • Reference: https://setuptools.pypa.io/en/latest/userguide/pyproject_config.html
• Provide default variable encoding parameters for xarray's to_netcdf #72
• Generalize hemisphere_to_global to be used for both numpy and xarray interface #48
  (to be cont.)

Incremental changes will first be merged to branch release1.0. On the release date, branch release1.0 will be merged to master.

Issues deploying package to PyPI channel

Procedures with packaging: https://packaging.python.org/en/latest/guides/distributing-packages-using-setuptools/#packaging-your-project

Using conda on mac to compile wheel would have this issue:

ld: unsupported tapi file type '!tapi-tbd' in YAML file

which is related to the fact that the conda python path is placed before the native path. I have not figured a way to get around it yet.

Make northern_hemisphere_results_only an instance variable of QGField

An error would arise if calling QGField.compute_reference_states(northern_hemisphere_results_only=True) and then compute_lwa_and_barotropic_fluxes() because northern_hemisphere_results_only=False is the default value of the method compute_lwa_and_barotropic_fluxes(). This can be fixed by making northern_hemisphere_results_only an instance variable (instead of local variable).

Thanks Nour-Eddine Omrani for reporting the issue.

Calculation using select pressure levels

Hi @csyhuang,

I am wondering if it is possible to calculate the LWA and the other fields not using all of the 37 pressure levels, but just a select few. For example, I am curious how would the fields look just by using the 4 levels around the jet stream (200, 225, 250 and 300 hPa). I have made some adjustments to the kmax parameter, but I am bit confused about the height adjustments as the first and last height won't be at the surface and top of course.

I have modified kmax from 49 to 12 as that was the suggestion in the error after I started the code using the wind and temperature fields at these select 4 levels, but then I just got an error for a segmentation fault. Do you have any suggestions on how to adjust it?

I am using a script based on demo_script_for_nh2018.ipynb

Martin

QGDataset does not support data input with even number of latitude gridpoints

When climate data with even number of latitudinal gridpoints is input to QGFieldBase, the following will occur inside the class:

hn2016_falwa/hn2016_falwa/oopinterface.py

Lines 286 to 292 in 9acd810

	if (ylat.size % 2 == 0) & (sum(ylat == 0.0) == 0):
	# Even grid
	self.need_latitude_interpolation = True
	self.ylat_no_equator = ylat
	self.ylat = np.linspace(-90., 90., ylat.size+1, endpoint=True)
	self.equator_idx = \
	np.argwhere(self.ylat == 0)[0][0] + 1

When returning the computation results to the user, the computed fields are interpolated back to the original grid (self.ylat_no_equator).

Right now, this behavior is not mirrored in QGDataset, which causes an inconsistency in results output when input data has even number of latitude gridpoints.

TODO: Modify QGDataset.interpolate_fields to handle this scenario.

Include the option to change the latitude domain for Uref inversion

The current version of hn2016_falwa.oopinterface.QGField is inverting the elliptic equation to solve Uref in the hemispheric domain (0°-90°N/S). An optional input boundary_lat will be added to this interface such that Uref can be inverted in a smaller hemispheric domain to avoid unrealistic Uref in model data analysis.

Thanks @EbrahimNabizadeh for raising the issue.

A bug in indexing in compute_reference_states.f90

There is an error in indexing at line 60 of the script - tbar should have dimension (jd,kmax). Will update this weekend to fix the error. Thanks Rui Zhang for reporting.

header for the compute_reference_states.f90 function

Dear Clare,

thank you for making all functions related to the LWA computation available on git! I am planning to compute the reference state u_ref, theta_ref according to Solomon and Nakamura 2010 part 1 and Huang and Nakamura 2017 and I'd like to use your function. My question is: could you please write a short header/comment for this function explaining the input arguments and what arguments are returned? There are some parameters I cannot guess what they mean (I also don't know fortran very well...).

Many thanks!

Paolo

Setup requirement.txt and virtual env config for non-anaconda users

The instructions and configurations listed in hn2016_falwa/examples/ to run the sample codes are specific to conda users. It would be great to include instructions for pip/virtualenv users as well, as many scientists are actually using that.

Add unit tests that cover xarray interface

See #45 for details.

Add unit tests that cover the xarray interface for release 0.7.0.

Bug in utilities.compute_qgpv_givenvort (v0.2.0)

A resolution of nlat=120 is assumed. I replace the hemispheric boundary with index nhalf=int(nlat/2). Bug fixed in test branch.

Equivalent latitude shall not assume nlat_s = 61

This is the default value I set, but there should be no values assumed.

Diagnostics (+ interpretations) to incorporate into MDTF

The goal to incorporate Finite-amplitude wave activity (FAWA) formalism into MDTF repo is to provide diagnostic tools for model comparison. Each diagnostic shall associate with an interpretation related to the physical process.

Below are some proposed candidates and associated physical interpretations:

Variable	Interpretation
Seasonal climatology of zonal mean FAWA	It quantifies the strength of vertically propagating planetary waves, and also relative strength of synoptic eddies in troposphere
Seasonal climatology of Uref	To quantify strength of mean-flow interaction, i.e., adiabatic adjustment of flow from an eddy-free reference state is given by Uref-Ubar(zonal mean wind)
Seasonal climatology of <LWA> (<...> = vertically averaged)	To quantify geographical distribution (and amplitude) of eddies
Seasonal climatology of temporal Covariance of <LWA> and <U> (as in NH18 Fig. 2A)	A measure of strength of nonlinear wave-mean flow interaction (via nonlinear zonal wave activity flux)

If you have any suggestions, please feel free to share here.

errors found for calculation of Uref with even N latitudes

Hi @csyhuang ,

First of all thanks for a very useful and clean code :)
I recently found an issue with the calculation of the LWA and Uref when the number of the latitudes is even. My input file's dimension look like this:

netcdf ATM_AlF50_F99_u_2051_jan {
dimensions:
lon = 192 ;
lat = 96 ;
lev = 39 ;
time = UNLIMITED ; // (30 currently)

The errors that I got was coming from this subroutine: hn2016_falwa/hn2016_falwa/Solve_URef_noslip_hemisphere.py
with the following errors:

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-2-45dc1438e9d4> in <module>()
     78        DT = PT3[-2,:,:].mean(axis=-1) - PT_ref
     79 
---> 80        u_MassCorr_noslip[sec_days,:,:],u_Ref_noslip[sec_days,:,:],T_MassCorr_noslip[sec_days,:,:],T_Ref_noslip[sec_days,:,:] =        Solve_Uref_adiabatic(str(time_array[sec_days]),UU3.mean(axis=-1),LWA_cos[:,:,:].mean(axis=-1),                            ylat,                            ephalf,                            DT,PT3.mean(axis=-1),                            np.random.rand(nlat),np.random.rand(nlat),use_real_Data=True)
     81 
     82 

/home/slubis/REF_final/June28_solve_uREF8_bothBC_editing.pyc in Solve_Uref_adiabatic(tstamp, zmum, FAWA_cos, ylat, ephalf2, Delta_PT, zm_PT, Input_B0, Input_B1, use_real_Data)
    131         # --- Interpolation of Delta_PT ---
    132         f_DT_toGaussian = interpolate.interp1d(ylat[:],Delta_PT[:], kind='linear')    # This is txt in Noboru's code
--> 133         Delta_PT1[:] = f_DT_toGaussian(alat_2[:])
    134         #Delta_PT1[0] = Delta_PT[0]
    135         #Delta_PT1[-1] = Delta_PT[-1]

/home/slubis/Enthought/Canopy_64bit/User/lib/python2.7/site-packages/scipy/interpolate/polyint.pyc in __call__(self, x)
     78         """
     79         x, x_shape = self._prepare_x(x)
---> 80         y = self._evaluate(x)
     81         return self._finish_y(y, x_shape)
     82 

/home/slubis/Enthought/Canopy_64bit/User/lib/python2.7/site-packages/scipy/interpolate/interpolate.pyc in _evaluate(self, x_new)
    588         y_new = self._call(self, x_new)
    589         if not self._extrapolate:
--> 590             below_bounds, above_bounds = self._check_bounds(x_new)
    591             if len(y_new) > 0:
    592                 # Note fill_value must be broadcast up to the proper size

/home/slubis/Enthought/Canopy_64bit/User/lib/python2.7/site-packages/scipy/interpolate/interpolate.pyc in _check_bounds(self, x_new)
    620                 "range.")
    621         if self.bounds_error and above_bounds.any():
--> 622             raise ValueError("A value in x_new is above the interpolation "
    623                 "range.")
    624 

ValueError: A value in x_new is above the interpolation range.

Thanks Clare ! :)

Best,
Sandro

Problem using qgfield_object.compute_lwa_and_barotropic_fluxes()

Hi
This is Amin Fazl Kazemi
I have tried to use calculate fluxes function. I downloaded global ERA5 data on 37 pressure levels, coarsened it to 1.25 degrees, passed it to the program and managed performing qgfield_object.compute_reference_states.
By the way when I pass qgfield_object to compute compute_lwa_and_barotropic_fluxes function, I face a problem :

ValueError: failed in converting 5th argument `qref' of compute_lwa_and_barotropic_fluxes.compute_lwa_and_barotropic_fluxes to C/Fortran array

I wonder the reason. Could you please clarify what I should do. of course the most difficult part (calculating referrence states( are already done, so many many thanks, but as for the addvection terms, I am in trouble. I can send you the file to test is needed.

regards

Fail to install

Hi Clare,

After typing
python setup.py develop
The error message shows:

Couldn't find index page for 'pytest-runner' (maybe misspelled?)
No local packages or download links found for pytest-runner
Traceback (most recent call last):
File "setup.py", line 78, in
zip_safe=False
File "/usr/lib64/python2.7/site-packages/numpy/distutils/core.py", line 186, in setup
return old_setup(**new_attr)
File "/usr/lib64/python2.7/distutils/core.py", line 112, in setup
_setup_distribution = dist = klass(attrs)
File "/usr/lib64/python2.7/site-packages/numpy/distutils/numpy_distribution.py", line 14, in init
Distribution.init(self, attrs)
File "/usr/lib/python2.7/site-packages/setuptools/dist.py", line 265, in init
self.fetch_build_eggs(attrs.pop('setup_requires'))
File "/usr/lib/python2.7/site-packages/setuptools/dist.py", line 289, in fetch_build_eggs
parse_requirements(requires), installer=self.fetch_build_egg
File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 618, in resolve
dist = best[req.key] = env.best_match(req, self, installer)
File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 862, in best_match
return self.obtain(req, installer) # try and download/install
File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 874, in obtain
return installer(requirement)
File "/usr/lib/python2.7/site-packages/setuptools/dist.py", line 339, in fetch_build_egg
return cmd.easy_install(req)
File "/usr/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 617, in easy_install
raise DistutilsError(msg)
distutils.errors.DistutilsError: Could not find suitable distribution for Requirement.parse('pytest-runner')

V

IMPORTANT bugfix release 0.7.2 + corrigendum

Two weeks ago, we discovered that there is a mistake in the derivation of expression of nonlinear zonal advective flux term:
$$\frac{1}{a\cos\phi} \frac{\partial}{\partial \lambda} \int^{\Delta\phi}_0 u_e q_e \cos(\phi+\phi^\prime) d\phi^\prime$$

The correct definition of $u_e$ in the integral should be:

$$u_e(\lambda, \phi, \phi^\prime, z, t) \equiv \omega_e a \cos\phi = \frac{u(\lambda, \phi + \phi^\prime, z, t)\cos\phi}{\cos(\phi+\phi^\prime)} - u_{\text{REF}}(\phi, z, t)$$

where $\omega = \frac{u}{a\cos\phi}$ is the angular velocity (and $\phi^\prime \in [0, \Delta\phi]$).

The fix has been done in the following commit and will be published in release 0.7.2:

1995e1d#diff-7e797030d6c037af018d523e2e6a20f0d9f01c3dd8f41a2d7f044b4800684455

This error led to an underestimation of nonlinear component of advective flux ("ua2" in the code). Fixing this error indeed makes traffic jam mechanism (i.e. how waves are slowed down by the nonlinear effect that grows with wave amplitude) more robust.

We will submit corrigendums for Neal et al. (2022, GRL) and Nakamura and Huang (2018, Science) to update the numerical results. The correct derivation of the flux expression can be found in the corrected supplementary materials of NH18. There is no change in conclusions in any of the articles.

Below are previews of updated figures (scale not adjusted yet) in Neal et al. (2022, GRL) and Nakamura nad Huang (2018, Science) due to this fix:

gfortran compile error hn2016_falwa/interpolate_fields.f90

Dear C. Huang,

I get an error like:
'''Warning: Unused variable ‘ylat’ declared at (1) [-Wunused-variable]
FATAL:/opt/local/bin/../libexec/as/x86_64/as: I don't understand 'm' flag!
error: Command "/usr/local/bin/gfortran -Wall -g -fno-second-underscore -m64 -fPIC -O3 -funroll-loops -Ibuild/src.macosx-10.6-x86_64-2.7/build/src.macosx-10.6-x86_64-2.7 -I/Users/semvijverberg/anaconda2/lib/python2.7/site-packages/numpy/core/include -I/Users/semvijverberg/anaconda2/include/python2.7 -c -c hn2016_falwa/interpolate_fields.f90 -o build/temp.macosx-10.6-x86_64-2.7/hn2016_falwa/interpolate_fields.o" failed with exit status 1'''

The full output is in the .txt output. To do this installation I installed the gfortran compiler 'gfortran-8.1-bin.tar.gz'. I tried installing both the normal and the test version, but I get the same error. I do not know what my next step would be to solve/unravel this problem. I hope you can help me.

I am now expert, perhaps I am doing something wrong..

Thanks in advance!
Sem

interpolate_fields.f90_error.txt

Fix documentation page: content of one page not displayed

The documentation for the module hn2016_falwa/oopinterface.py is not displayed. When I generated that with Sphinx locally, the page is properly displayed as shown [here].(https://htmlpreview.github.io/?https://github.com/csyhuang/hn2016_falwa/blob/master/docs/build/html/Object%20Oriented%20Interface.html)

However, the page is broken on readthedocs.io:
https://hn2016-falwa.readthedocs.io/en/latest/Object%20Oriented%20Interface.html

I spent quite some time to try fixing it but futile. Anyone knowing what the issue is?

application to oceanic simulation

Hi Clare,

I am interested in your elegent theory and going to give a try in an idealized oceanic simulation.

This is a channel simulation (periodic in zonal direction) on an Beta-plane, a prototype model of Antarctic Circumpolar Current (ACC), which is also close to the context of classical baroclinic instability on midlatitudes.

My simulation is not QG but full dynamics with hydrostatic approximation in a depth coordinate, and definitely away from equator. I just wonder if the current python code (in QG approximation), tested in atmospheric context, can be adapted to diagnose the FALWA in this ocean simulation?

To-do for release 0.4.2

• Incorporate changes made by Christopher - Include instructions on how to remove old compiled fortran modules
• Remove download_data.py as ERA-I has deprecated
• Check if codecov is working (seems it stopped working)
• Check consistency between readme.md and documentation
• Add additional unit tests for oopinterface (sanity checks)
• Clean up sample scripts (in folder examples/)
• (to be updated)

Bug in GitHub action

The GitHub workflow .github/workflows/workflow.yml is using actions/setup-python@master but it seems cannot read double-digit python version. When I put in python-version: 3.10 it tested with python version 3.1.
Because of this, I am temporarily using python 3.9 in the file, just to get rid of the error caused by this.

hn2016_falwa/.github/workflows/workflow.yml

Line 8 in 7ff8f0c

python-version: [3.9]

hn2016_falwa/.github/workflows/workflow.yml

Line 17 in 7ff8f0c

python-version: 3.9

Translate all f2py modules into python/cython scripts

Communicated with MDTF liaison and found that cython is an acceptable solution for MDTF package. The current plan would be to translate all f2py modules into cython to eliminate the dependency of the package on Fortran Compiler.

Speed optimization will be addressed in another ticket.

Inconsistent normalization with prefactor in column average

Hello @csyhuang,

I think there is an inconsistency with the normalization in the default parameters for the column average. The density-weighted vertical average is defined as (Huang and Nakamura 2016):

In the implementation, the upper bound is dependent on data availability, so the normalization constant will be a bit smaller than H. The value of the normalization is currently given to QGField with the prefactor parameter, set to 6500 m by default here. The default number of vertical levels is 49 and levels are spaced 1000 m apart, so the upper bound of the integral should be 49 km. With the scale height set to 7000 m, that results in a normalization constant of 6994 m. So the default choice of prefactor seems inconsistent with the choices of the other parameters in the constructor of QGField (the right upper bound for 6500 m normalization is somewhere around 18.5 km).

If I'm not mistaken with the above, I'd suggest to either set prefactor to the scale height (more consistent, smaller error) or, even better, calculate prefactor in the constructor with a quick numerical evaluation of the normalization integral and remove it as a parameter. kmax and dz are enough to determine the appropriate value and the effort to evaluate the integral should be negligible.

Chris