liambindle / sparselt

A small library for regridding Earth system data with vectorized sparse linear transforms

sparselt

sparselt is a small library for regridding Earth system model data. More generally, sparselt makes it easy to apply vectorized sparse linear transforms to high-dimensionality datasets.

Operations like regridding can be written as a linear transform. The matrix has shape (M,N) where N is the flat-size of the output grid and M is the flat-size of the input grid. Tools like ESMF_RegridWeightGen can generate these "regridding weights" for a variety of methods (e.g., bilinear, conservative, etc.) for a variety of grid types (e.g., lat-lon, gaussian, cubed-sphere, stretched cubed-sphere, etc.). sparselt is a small formalization for applying sparse transforms like this.

sparselt is intended to be used in combination with

• gridspec for generating grid definition files
• ESMF and specifically the ESMF_RegridWeightGen tool
• xarray for working with NetCDF datasets

The workflow for using sparselt for regridding is

1. Create grid definitions files for the input and output grids using a tool like gridspec
2. Generate regridding weights using a tool like ESMF_RegridWeightGen
3. Apply the weights to your dataset with sparselt

What about xESMF? xESMF is a terrific for regridding logically rectangular grids, however, it has limited support for higher-order grids like cubed-sphere grids (see here).

Installation

For the meantime, you can install sparselt like so

$ pip install git+https://github.com/LiamBindle/sparselt.git 

Examples

See the examples/ subdirectory for an example, and examples/sample_data/README.md for a description of how the sample data files were generated.

Applying ESMF regrid weights

The following is an example of regridding a C48 dataset to a lat-lon grid using weights generated by ESMF_RegridWeightGen.

import xarray as xr
import sparselt.esmf
import sparselt.xr

# Load dataset on C48 grid
ds = xr.open_dataset('my_c48_dataset.nc')

# Create a linear transformer object from an ESMF weights file
transform = sparselt.esmf.load_weights(
    'regrid_weights.nc',                               # Generated by ESMF_RegridWeightGen
    input_dims=[('nf', 'Ydim', 'Xdim'), (6, 48, 48)],  # applied along input dimensions
    output_dims=[('lat', 'lon'), (90, 180)],           # result dimensions
)

# Apply the transform to ds
ds = sparselt.xr.apply(transform, ds)

Initially, the dataset looks like this:

<xarray.Dataset>
Dimensions:          (Xdim: 48, Ydim: 48, lev: 24, nf: 6, time: 1)
Coordinates:
  * Xdim             (Xdim) float64 305.8 307.4 309.0 ... 31.01 32.63 34.21
  * Ydim             (Ydim) float64 -44.21 -42.62 -41.01 ... 41.01 42.62 44.21
  * lev              (lev) float64 1.0 2.0 3.0 4.0 5.0 ... 21.0 22.0 23.0 24.0
  * nf               (nf) int32 1 2 3 4 5 6
  * time             (time) datetime64[ns] 2019-09-16
Data variables:
    SpeciesConc_CO   (time, lev, nf, Ydim, Xdim) float32 ...
    SpeciesConc_NO2  (time, lev, nf, Ydim, Xdim) float32 ...
    SpeciesConc_O3   (time, lev, nf, Ydim, Xdim) float32 ...

After regridding, the dataset looks like this:

<xarray.Dataset>
Dimensions:          (lat: 90, lev: 24, lon: 180, time: 1)
Coordinates:
  * lev              (lev) float64 1.0 2.0 3.0 4.0 5.0 ... 21.0 22.0 23.0 24.0
  * time             (time) datetime64[ns] 2019-09-16
Dimensions without coordinates: lat, lon
Data variables:
    SpeciesConc_CO   (time, lev, lat, lon) float64 6.006e-08 ... 1.06e-07
    SpeciesConc_NO2  (time, lev, lat, lon) float64 4.517e-15 ... 2.317e-12
    SpeciesConc_O3   (time, lev, lat, lon) float64 2.247e-08 ... 4.854e-08

General sparse linear transforms

Below is an example of apply a general sparse linear transform (e.g., could be extended for using weights generated by an alternative tool, 3D regridding, vector regridding, 4D spacetime sampling, etc.).

import xarray as xr
import sparselt
import sparselt.xr

# Open input dataset
ds = xr.open_dataset('my_input_data.nc')

# Load weights from file
ds_weights = xr.open_dataset('my_weights.nc')

# Get sparse matrix elements
weights = ds_weights.S
row_ind = ds_weights.row
col_ind = ds_weights.col

# Create a linear transformation object
transform = sparselt.SparseLinearTransform(
    weights, row_ind, col_ind,
    input_transform_dims=[('lat', 'lon'), (361, 576)], 
    output_transform_dims=[('nf', 'Ydim', 'Xdim'), (6, 180, 180)],
    one_based_indices=True,
    order="C",
)

# Apply the transformation
ds = sparselt.xr.apply(transform, ds[['tas', 'pr']])

sparselt's Issues

Can't do sparselt.xr.apply if the input dataset's dimensions have the same name as the template dataset

Reported by Joonhee.

The data I would like to regrid is 0.5x0.625 deg (361 lat, 576 lon) and I’d like to regrid this to 4x5 deg (46 lat, 72 lon). This is what the source data looks like below:

Steps:

1. Generate grid description file for source

$ gridspec-create latlon 361 576 -pc -dc

2. Generate grid descrption file for destination

$ gridspec-create latlon 46 72 -pc -dc

3. Generate weights for transformation

$ ESMF_RegridWeightGen -s regular_lat_lon_361x576.nc -d regular_lat_lon_46x72.nc -m conserve -w regrid_weights_latlon361x576_to_ latlon46x72.nc

4. Apply transformation

# Load dataset
ds = xr.open_dataset("AEIC_20190101.0.5x0.625.36L.nc")
print('Original dataset: {}\n\n'.format(str(ds)))

# Create SparseLinearTransform object from an ESMF weights file
transform = sparselt.esmf.load_weights(
    'regrid_weights_latlon361x576_to_ latlon46x72.nc',
    input_dims=[('lat', 'lon'), (361, 576)],
    output_dims=[('lat', 'lon'), (46, 72)]
)

# Open an output template dataset (optional)
output_template = xr.open_dataset('regular_lat_lon_46x72.nc')

# Apply the transform to ds
ds = sparselt.xr.apply(transform, ds, output_template)

print('Transformed dataset: {}\n\n'.format(str(ds)))

This is giving me an error at the sparselt.xr.apply step as “ValueError: inconsistent size for core dimension 'lat': 46 vs 361”.

Question: can this be incorporated into xESMF?

What about xESMF? xESMF is a terrific for regridding logically rectangular grids, however, it has limited support for higher-order grids like cubed-sphere grids (see here).

Are there any limitations that would prevent incorporating this into xESMF so that its support for cubed-sphere grids is expanded? Using xESMF is useful because it keeps the regridding tools all in one place.

Error while regridding a file

Hi,

Good morning. I have been trying to use the sparselt library to regrid a C24 resolution GCHP output into 4x5 degree output. I was successful in creating "c24_gridspec.nc", "regular_lat_lon_45x72.nc", and "esmf_regrid_weights_c24_to_latlon45x72.nc" files. However, I am getting the following error when trying to apply the sparset library using the commands:

import xarray as xr
import sparselt.esmf
import sparselt.xr

ds = xr.open_dataset('trial.nc')
transform = sparselt.esmf.load_weights(
    'esmf_regrid_weights_c24_to_latlon45x72.nc',                     
    input_dims=[('nf', 'Ydim', 'Xdim'), (6, 24, 24)],  
    output_dims=[('lat', 'lon'), (45, 72)],        
)
ds = sparselt.xr.apply(transform, ds)

ValueError: operand to apply_ufunc has required core dimensions ['nf', 'Ydim', 'Xdim'], but some of these dimensions are absent on an input variable: ['nf', 'Ydim', 'Xdim']

The dimension of my C24 GCHP file (trial.nc) looks ok to me. Do you have any insight into what might cause this issue? Unfortunately, I could not find a way to attach the NetCDF file ( trial.nc) for your consideration. I would highly appreciate it if you could share your suggestion on it. Thanks.

Question: regridding stretched grid

Hi Liam--

I'm trying to re-grid some stretched grid simulations and am having a bit of a hard time figuring out how to do so since the input dimensions change, so its not clear to me how to input an x and y dim. I was wondering if this is possible with sparselt, or if there is a better way to work with stretched grid outputs?

Thanks!