nawendt / esmpy-tutorial

Basic tutorial for ESMPy Python package

esmpy-tutorial

Basic tutorial for ESMPy Python package

ESMPy is the Python interface for the Earth System Modelling Framework (ESMF) regridding utility. You can find out more here. This repository contains a brief tutorial on how you can use this package to do some common regridding tasks. These examples use data from numerical weather prediction models as well as surface stations, all which are included. Take a look at the Jupyter Notebook and see how to use the package. Happy regridding!

Current examples available:

• grid to grid (bilinear)
• grid to grid (first-order conservative)
• points to grid
• grid to points
• parallel grid to grid via MPI (bilinear)

Plots in this tutorial are created using cartopy. Previous versions were created using basemap. A branch containing the old basemap version is around for now, but will be deleted at some point.

Running the parallel regrid example

Replace NPROCS with the number of processes you want to use and be sure mpi4py is installed.

mpirun -n NPROCS python esmpy_mpi_example.py

Questions about this tutorial? Have an example you would like to see? Contact me or submit an issue and I'll see what I can do to help. If you have specific questions about the ESMPy software or its development, you will be much better served by getting into contact with the actual ESMPy developers (I am not one, just a happy user) at [email protected].

esmpy-tutorial's Issues

Example maps should be done using cartopy

Several deprecation warnings have appeared for the basemap methods used in the tutorial. Furthermore, the development on basemap has slowed down considerably. It makes much more sense at this point to use cartopy as the library has been gaining steam and is under very active development. Going forward, cartopy will likely replace basemap as the standard mapping library for Python.

Use more efficient memory layout

First thanks for this clear tutorial! It helped a lot when I first started to write my xESMF package.

In your current example, a normal numpy array (C-ordered) is passed to ESMPy (expects Fortran-ordered), so it incurs overhead for rearranging the memory layout. It is much better to pass the transpose of an array (changes from C-ordered to Fortran-ordered) to ESMPy. This would make passing arrays ~10x faster. See ESMPy_memory_layout.ipynb for more explanations and timing.

In my xESMF timing test, passing the transpose (Fortran-ordered) is 20x faster than passing a normal C-ordered array.

Using transpose also allows periodic boundary conditions to work correctly. ESMPy's periodic boundary option requires a shape of [Nlon, Nlat], but a typical numpy array has the shape [Nlat, Nlon]. See ESMPy_periodic_fix_ordering.ipynb

ESMPy for curvilinear fine scale to horizontal coarse downscaling

Hello:

I adapted your tutorial code to downscale a 5424 x 5424 curvilinear grid conservatively to a 162 (1 degree lat) by 128 (1.25 degree lon) section of a coarse global model grid.

The code below worked quite well for a small test where I downscaled a 170 x 170 grid to 162 x 128. But for the larger scale grid I got the following error when trying to run with mpirun and 4 processors. I know you are a user and not support but any help would be greatly appreciated.

ERROR
Primary job terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.

---

mpirun noticed that process rank 2 with PID 0 on node ldt-4295279 exited on signal 9 (Killed).

CODE:
import ESMF
ESMF.Manager(debug=True)
import numpy as np
from netCDF4 import Dataset

glm_nc = Dataset('glmlatlon_goes16_with_glm_ff_var_no_time.nc')

glm_ff = glm_nc['glm_ff'][:,:]
glm_ff_shape = glm_ff.shape
print(glm_ff_shape)
glm_lats = np.linspace(1, 5424, 5424)
print(glm_lats)
print(len(glm_lats))
glm_lat_b = np.linspace(0, 5425, 5425)
print(glm_lat_b)

glm_lons = np.linspace(1, 5424, 5424)
print(glm_lons)
print(len(glm_lons))
glm_lon_b = np.linspace(0, 5425, 5425)
print(glm_lon_b)

glm_lat_corners,glm_lon_corners = np.meshgrid(glm_lat_b, glm_lon_b)

c96_nc = Dataset('c96_grid_May2019_region_0_to_180_no_time.nc')

c96_grid = c96_nc['flash_freq'][:,:]

c96_grid_shape = c96_grid.shape
print(c96_grid_shape)

latitudes = c96_nc['lat'][:]
lat_corners = np.linspace(-81.5, 81.5, 163)
print(lat_corners)
longitudes = c96_nc['lon'][:]

lon_corners = np.linspace(-155.5, 5.5, 129)

print(lon_corners)

c96_corner_lat, c96_corner_lon = np.meshgrid(lat_corners, lon_corners)

FORTRAN_CONTIGUOUS = True

sourcegrid = ESMF.Grid(np.array(glm_ff_shape),staggerloc=[ESMF.StaggerLoc.CENTER, ESMF.StaggerLoc.CORNER],
coord_sys=ESMF.CoordSys.SPH_DEG) #, staggerloc=ESMF.StaggerLoc.CENTER)
destgrid = ESMF.Grid(np.array(c96_grid_shape),staggerloc=[ESMF.StaggerLoc.CENTER, ESMF.StaggerLoc.CORNER],
coord_sys=ESMF.CoordSys.SPH_DEG) #, staggerloc=ESMF.StaggerLoc.CENTER)

source_lon = sourcegrid.get_coords(0, staggerloc=ESMF.StaggerLoc.CENTER)
source_lat = sourcegrid.get_coords(1, staggerloc=ESMF.StaggerLoc.CENTER)

dest_lon = destgrid.get_coords(0, staggerloc=ESMF.StaggerLoc.CENTER)
dest_lat = destgrid.get_coords(1, staggerloc=ESMF.StaggerLoc.CENTER)

source_lon[...] = source_lon
source_lat[...] = source_lat

dest_lon[...] = dest_lon
dest_lat[...] = dest_lat

glm_lat = sourcegrid.get_coords(1, staggerloc=ESMF.StaggerLoc.CORNER)
glm_lon = sourcegrid.get_coords(0, staggerloc=ESMF.StaggerLoc.CORNER)

c96_lat = destgrid.get_coords(1, staggerloc=ESMF.StaggerLoc.CORNER)
c96_lon = destgrid.get_coords(0, staggerloc=ESMF.StaggerLoc.CORNER)

if FORTRAN_CONTIGUOUS:
glm_lat[...] = glm_lat_corners.T
glm_lon[...] = glm_lon_corners.T

c96_lat[...] = c96_corner_lat.T
c96_lon[...] = c96_corner_lon.T

else:
glm_lat[...] = glm_lat_corner
glm_lon[...] = glm_lon_corner

c96_lat[...] = c96_corner_lat
c96_lon[...] = c96_corner_lon

sourcefield = ESMF.Field(sourcegrid, name='GLM Flash Frequency')

destfield = ESMF.Field(destgrid, name='Downscaled GLM Flash Frequency')

if FORTRAN_CONTIGUOUS:
sourcefield.data[...] = glm_ff.T
else:
sourcefield.data[...] = glm_ff

regrid = ESMF.Regrid(sourcefield, destfield, regrid_method=ESMF.RegridMethod.CONSERVE, unmapped_action=ESMF.UnmappedAction.IGNORE)

destfield = regrid(sourcefield, destfield)

glm_downscaled = destfield.data

print(glm_downscaled)

np.savetxt("glm_downscale_052019.csv", glm_downscaled, fmt="%1.14f", delimiter=",")

---

Does ESMpy support regridding using weights?

Hi,

Does ESMpy support regridding using weights? ESMF does has, but I didnot find any information in ESMpy.

Thanks!

Grid projections need to be handled correctly

The NAM and RUC grids used in the tutorial have LCC projection coordinates. What is confusing is that the coordinates that are extracted from the grids are latitude and longitude, which typically means you are working with geographic coordinates; however, the latitude and longitude values are simply converted from projection space. In order for the regridding to be done correctly, the latitude and longitude must be converted back to projection space and ESMF.CoordSys.CART must be used instead of ESMF.CoordSys.SPH_DEG.