Inconsistent dimension name for string length between gas optics NetCDF files about rte-rrtmgp

[RobertPincus]

@pernak18 I'm hoping you can use some Python-fu here? Roughly we want a PR with a new SW coefficients file that has precisely the same structure but a revised dimension name consistent with the LW file.

[pernak18]

@RobertPincus sounds doable, but will have to wait until i have some free time. i'm a pretty small line item in Eli's budget for April

[pernak18]

problem was more complex than i thought it would be, but the code i used at NERSC (cori) is:

#!/usr/bin/env python

import os, sys

PIPPATH = '{}/.local/'.format(os.path.expanduser('~')) + \
    'cori/3.7-anaconda-2019.10/lib/python3.7/site-packages'
paths = [PIPPATH, '../common']
for path in paths: sys.path.append(path)

import xarray as xa
import numpy as np

strVars = ['gas_minor', 'gas_names', 'identifier_minor', 
           'minor_gases_lower', 'minor_gases_upper', 
           'scaling_gas_lower', 'scaling_gas_upper']

ncFile = 'rrtmgp-data-sw-g224-2018-12-04.nc'

outDS = xa.Dataset()
with xa.open_dataset(ncFile, concat_characters=False) as ds:
    outDS.attrs = ds.attrs
    ncVars = list(ds.keys())
    for ncVar in ncVars:
        inDat = ds[ncVar]
        if ncVar in strVars:
            inDat = inDat.astype(str)
            strings = [''.join(string) for string in inDat.values]
            outDS[ncVar] = xa.DataArray(
                np.array(strings, dtype=np.dtype(('S', 32))), 
                dims=[inDat.dims[0]], attrs=inDat.attrs)
        else:
            outDS[ncVar] = ds[ncVar]
        # endif ncVar
    # end ncVar loop
# endwith
    
# https://github.com/pydata/xarray/issues/2899
# workaround is admittedly not the solution we want (arrays end up being strings)
# but it leads to https://github.com/pydata/xarray/issues/3407#issuecomment-542594876
# which solves our problem and is what was implemented
outNC = 'modified_{}'.format(ncFile)
encode = {}
for sv in strVars: 
    encode[sv] = {'zlib':True, 'complevel':5, 'char_dim_name': 'string_len'}
outDS.to_netcdf(outNC, encoding=encode, format='NETCDF3_CLASSIC')

print('Wrote {}'.format(outNC))

i apparently cannot attach the resulting file in this issue. do we want this code somewhere in version control, or does the code snippet suffice?

[RobertPincus]

This seems awesome, @pernak18, thanks so much. Can you open a PR against develop in which you update rrtmgp/data/rrtmgp-data-sw-g224-2018-12-04.nc to use the file with the renamed dimension? Having the code in the issue and/or PR is enough, I think.

[pernak18]

for my reference:

% pwd
/global/homes/p/pernak18/RRTMGP/rte-rrtmgp
% git branch rp_dev
% git checkout rp_dev
Switched to branch 'rp_dev'
% cd rrtmgp/data/
% cp ~/RRTMGP/g-point-reduction/dim_swap/modified_rrtmgp-data-sw-g224-2018-12-04.nc rrtmgp-data-sw-g224-2018-12-04.nc 
cp: overwrite 'rrtmgp-data-sw-g224-2018-12-04.nc'? y

on cori.

needed my own branch before creating Pull Request (I don't have permissions on develop).

[Chiil]

The files in the new main repository have inconsistent NetCDF types:

chiel@MacBookChiel:~/devel/radiation/rte-rrtmgp/rrtmgp/data$ ncdump -k rrtmgp-data-sw-g224-2018-12-04.nc 
netCDF-4
chiel@MacBookChiel:~/devel/radiation/rte-rrtmgp/rrtmgp/data$ ncdump -k rrtmgp-data-lw-g256-2018-12-04.nc 
classic

[RobertPincus]

thanks for noticing

[pernak18]

@RobertPincus does this mean we want to switch LW to netCDF-4 or SW to classic?

[RobertPincus]

@pernak18 I'm not sure it matters, netCDF-4 is the future though. @Chiil do you have an opinion?

[m214089]

HI,

I do not think that netcdf-4 is the future after the hdf group changed there working model. I think for such small files the netcdf classic format (aka netcdf-3 is best.
Cheerio,
Luis

[pernak18]

i've converted SW to classic. xarray has 4 different options, and i initially went with NETCDF4_CLASSIC as a combination of both but then decided to stay consistent with the LW, so:

outDS.to_netcdf(outNC, encoding=encode, format='NETCDF3_CLASSIC')

[RobertPincus]

Fixed in commit cd7a6c2 (#109)