joleenf / clavrx_merra2

Update CLAVR-x MERRA code for reading/writing MERRA2 data as well as other models

1. You may see a warning of the form:

"UserWarning: WARNING: valid_range not used since it cannot be safely cast to variable data type"

This is not a fatal warning. when python pulls data from the netCDF file during
the enumerate() function, the .valid_range attribute, used for data mask, is not understood
and therefore is not applied. Since the time information in the enumerate()
should not be masked to begin with, this shouldn't be a concern.

2. FRSEAICE (sea ice-fraction) is the output 'ice fraction' variable, defined in the
   FLX file around Lines 235-241 of merra24clavrx.py. This is done because in comparison to GFS .hdf
   files, it appears the 'ice fraction' variable from GFS is only sea-ice fraction. In addition, the
   MERRA2 ice-fraction variable (FRACI) appears to be broken with fill-values over the ocean, the
   entirety of Antarctica, and most of Greenland. This variable is being collected and passed along to
   .hdf output as 'FRACI' for later fix/use.

3. Input files are retrieved within the python code using scripts/wget_all.sh Please see [ How to Download Data Files from HTTPS Service with wget ](https://disc.gsfc.nasa.gov/information/howto?keywords=Wget&title=How%20to%20Download%20Data%20Files%20from%20HTTPS%20Service%20with%20wget)

The script calls nine wget scripts but some of the nine files are not currently being used by /clavrx-merra2/merra2/merra24clavrx.py to produce an output file. The
current formulation does not make use of the inst6_3d_ana_Nv file, which contains analysis data on
hybrid sigma-pressure coordinate surfaces. This file was intended to provide the data requested at
the 0.9950 sigma-level, or a level close to it, but it looks like the 10m data is being substituted
and it's working fine.

4. Currently running the program for one year/month/day at a time using merra24clavrx.py The code needs the appropriate conda environment from merra2_clavrx.yml

    conda activate merra2_clavrx```
  
  
After running the run-script for a selected INPUT_DATE, files will appear in /tmp/out/YYYY/. This  
directory is currently being removed every time the run-script is used, so if you iterate over  
multiple dates, you'll scrub the output on every intermediate date and only have the output for the  
last date. You can get rid of this by commenting-out Lines 24-27 of test_merra24clavrx.sh.


JMF NOTES:
Use test_merra24_brett.sh to test original code.

clavrx_merra2's Issues

fill value incorrect/sets wrong attribute fill

clavrx_MERRA2/merra2/merra24clavrx_brett.py

Lines 500 to 510 in 86ac237

	elif self.out_name == 'rh':
	temp_sds = in_sds.variables['T'] # temperature in [K] (Time, Height, Y, X)
	temp_k = temp_sds[time_ind]
	temp_k = _trim_toa(temp_k)
	fill = temp_sds._FillValue
	temp_k[temp_k == fill] = np.nan
	fill = in_sds.variables[self.in_name]._FillValue
	data[data == fill] = np.nan
	rh = qv_to_rh(data, temp_k)
	rh[np.isnan(rh)] = fill
	out_sds.set(_refill(_reshape(rh, self.ndims_out, fill), fill))

In this block, the fill value later written as an attribute to the hdf is set to 1e+15. However, the calculated field (data) is a masked_array with fill_value of 1e+20. This creates a mismatch when the mask array is filled before writing to the hdf.

Also, trim_toa is truncating the levels without checking range.