Prepare input data about speedyweather.jl

[samhatfield]

Examples:

[milankl]

While I've changed the default spectral resolution from T30 to T31, the grid space resolution remains unchanged (hence the same input data in /input_data can be used), there's only one more row of spectral coefficients. Currently the Gaussian latitudes are obtained for nlat = 48 with

julia> using FastGaussQuadrature
julia> nlat = 48
julia> reverse(asind.(FastGaussQuadrature.gausslegendre(nlat)[1]))
48-element Vector{Float64}:
  87.15909455586285
  83.47893666931716
  79.77704565482563
  76.07024446254513
  72.36158102934485
  68.65201678951749
  64.94194948875752
  61.231573188077135
  57.520993797969965
  53.810274031941425
  50.09945341298685
  46.38855811160542
  42.677606172604904
  38.966610469454025
  35.255580461368176
  31.54452328402167
  27.833444451993234
  24.122348326087984
  20.411238433567785
  16.700117693842667
  12.988988582088142
   9.277853251507858
   5.566713627913589
   1.8555714859932573
  -1.8555714859932573
  -5.566713627913589
  -9.277853251507858
 -12.988988582088142
 -16.700117693842667
 -20.411238433567785
 -24.122348326087984
 -27.833444451993234
 -31.54452328402167
 -35.255580461368176
 -38.966610469454025
 -42.677606172604904
 -46.38855811160542
 -50.09945341298685
 -53.810274031941425
 -57.520993797969965
 -61.231573188077135
 -64.94194948875752
 -68.65201678951749
 -72.36158102934485
 -76.07024446254513
 -79.77704565482563
 -83.47893666931716
 -87.15909455586285

whereby the asind is used instead of acosd to obtain latitudes not colatitudes - whichever way you want to look at it.

[samhatfield]

Oh, one other thing. What do you think about splitting up the input files so that it's one field per file? This would cut out one step when preparing the input data.

[samhatfield]

Yes, I think this should be more than adequate for now :p

[milankl]

I'll leave this issue open for other input data we'll need eventually.

[samhatfield]

See branch sh/input-data.

[samhatfield]

The Gaussian latitudes for the resulting T31 data do not match those for the currently used orography data:

>>> orig = iris.load("orog_actual_T30.nc")[0]; orig.coords("latitude")
[DimCoord(array([ 87.159,  83.479,  79.777,  76.07 ,  72.362,  68.652,  64.942,
        61.232,  57.521,  53.81 ,  50.099,  46.389,  42.678,  38.967,
        35.256,  31.545,  27.833,  24.122,  20.411,  16.7  ,  12.989,
         9.278,   5.567,   1.856,  -1.856,  -5.567,  -9.278, -12.989,
       -16.7  , -20.411, -24.122, -27.833, -31.545, -35.256, -38.967,
       -42.678, -46.389, -50.099, -53.81 , -57.521, -61.232, -64.942,
       -68.652, -72.362, -76.07 , -79.777, -83.479, -87.159]), standard_name='latitude', units=Unit('degrees'), long_name='latitude', var_name='lat')]
>>> new = iris.load("orog_T31.nc")[0]; new.coords("latitude")
[DimCoord(array([ 88.125,  84.375,  80.625,  76.875,  73.125,  69.375,  65.625,
        61.875,  58.125,  54.375,  50.625,  46.875,  43.125,  39.375,
        35.625,  31.875,  28.125,  24.375,  20.625,  16.875,  13.125,
         9.375,   5.625,   1.875,  -1.875,  -5.625,  -9.375, -13.125,
       -16.875, -20.625, -24.375, -28.125, -31.875, -35.625, -39.375,
       -43.125, -46.875, -50.625, -54.375, -58.125, -61.875, -65.625,
       -69.375, -73.125, -76.875, -80.625, -84.375, -88.125]), standard_name='latitude', units=Unit('degrees'), long_name='latitude', var_name='lat')]

I'm not sure what assumptions CDO's remapbil operator makes about the Gaussian latitudes. @milankl, if T31 will be the new default instead of T30, could you post here the latitudes that SpeedyWeather.jl computes for T31 so we can compare with what CDO gives us?

[samhatfield]

89dfe1d fixes this.

>>> orig = iris.load("orog_actual_T30.nc")[0]; orig.coords("latitude")
[DimCoord(array([ 87.159,  83.479,  79.777,  76.07 ,  72.362,  68.652,  64.942,
        61.232,  57.521,  53.81 ,  50.099,  46.389,  42.678,  38.967,
        35.256,  31.545,  27.833,  24.122,  20.411,  16.7  ,  12.989,
         9.278,   5.567,   1.856,  -1.856,  -5.567,  -9.278, -12.989,
       -16.7  , -20.411, -24.122, -27.833, -31.545, -35.256, -38.967,
       -42.678, -46.389, -50.099, -53.81 , -57.521, -61.232, -64.942,
       -68.652, -72.362, -76.07 , -79.777, -83.479, -87.159]), standard_name='latitude', units=Unit('degrees'), long_name='latitude', var_name='lat')]
>>> new = iris.load("orog_T31.nc")[0]; new.coords("latitude")
[DimCoord(array([ 87.15909456,  83.47893667,  79.77704565,  76.07024446,
        72.36158103,  68.65201679,  64.94194949,  61.23157319,
        57.5209938 ,  53.81027403,  50.09945341,  46.38855811,
        42.67760617,  38.96661047,  35.25558046,  31.54452328,
        27.83344445,  24.12234833,  20.41123843,  16.70011769,
        12.98898858,   9.27785325,   5.56671363,   1.85557149,
        -1.85557149,  -5.56671363,  -9.27785325, -12.98898858,
       -16.70011769, -20.41123843, -24.12234833, -27.83344445,
       -31.54452328, -35.25558046, -38.96661047, -42.67760617,
       -46.38855811, -50.09945341, -53.81027403, -57.5209938 ,
       -61.23157319, -64.94194949, -68.65201679, -72.36158103,
       -76.07024446, -79.77704565, -83.47893667, -87.15909456]), standard_name='latitude', units=Unit('degrees'), long_name='latitude', var_name='lat')]

Looks the same to me.

[samhatfield]

I could just add all of the resolutions to version control since they're not so big:

-rw-r----- 1 nash rd  31K Feb 25 13:10 orog_T31.nc
-rw-r----- 1 nash rd  45K Feb 25 13:14 orog_T42.nc
-rw-r----- 1 nash rd 143K Feb 25 13:14 orog_T85.nc
-rw-r----- 1 nash rd 529K Feb 25 13:14 orog_T170.nc
-rw-r----- 1 nash rd 2.1M Feb 25 13:14 orog_T341.nc
-rw-r----- 1 nash rd 8.1M Feb 25 13:14 orog_T682.nc
-rw-r----- 1 nash rd  13M Feb 25 11:18 orog_TCO1279.nc

What do you think? Then you could modify the transform test to test all resolutions.

[milankl]

Is this the orography in spectral space or in grid space corresponding to the T??? spectral truncation? I'd prefer grid space input files, given that we may change the spectral packing and/or normalisation (see #32). Furthermore, one, rather high resolution, should suffice, happy with one file per field too. But even at 8MB could you either compress them well or send them to me first?

7 mantissa bits should be fine, that makes a hypothetical Everest only 16m shorter and even for 64-128m height we'd still get 50cm accuracy - much higher than the horizontal resolution allows for anyway. I'm asking because I'd like to avoid getting a rather large repository (as all the history adds up to its size).

[samhatfield]

These are all in grid-point space. I'm happy to label them by the Gaussian grid convention if you prefer? E.g. orog_N24.nc for 96x48/T31 etc.?

Yes, so the only data kept in the repository could be the T682. Though then the user would need CDO to prepare input data for other resolutions, unless you find a Julian approach for this as well.

I'm not a skilled compressor I'm afraid - I could only get orog_T682.nc down to 6.4 MB :p I'll send it to you.

Edit: actually I've already committed the original orography file orog_TCO1279.nc to my branch so you could just try generating the others and compressing them from there.

[milankl]

Just checked for the TCO1279 resolution there's with 7 mantissa bits still 99.9% of information. So happy to chop it down to that if you can? I get the 13MB file then down to 5MB with lossless, if you can switch on Zlib compression in netcdf that'd be dope! For T682 it's probably then only a 2MB file...

[milankl]

Yes, so the only data kept in the repository could be the T682. Though then the user would need CDO to prepare input data for other resolutions, unless you find a Julian approach for this as well.

Just the highest resolution T682 is technically fine as we already have an interpolation in spectral space going, so we can just coarsen the field at initialisation.

[samhatfield]

Good point. So CDO is not actually needed. In that case we might as well forget the whole prepare_input_data directory and just get SpeedyWeather to do everything from the T682 data. We'll keep the TCO1279 orography somewhere in case we need to go higher than T682, but keep it out of the repository.

I'll send you the TCO1279 and T682 data on Slack.

[milankl]

Got the T682/2048x1024 data from 8MB down to about 2.5MB using nco

ncks --baa=8 --ppc orog=7 orog_T682.nc orog_T682_round.nc

(baa is the bitrounding mode, orog=7 retains 7 mantissa bits for the variable orog, lossless is on by default in ncks)

Thanks, Sam. PR coming!

[samhatfield]

Excellent!

[milankl]

Pull request: #41 uses this file and automatically truncates to the desired resolution. Code works with T<=682 for higher we'd need to pad the spectral coefficients with zero, but T682 already gives us a resolution of ~20km, I doubt we would want to go higher?!

[milankl]

@samhatfield the input data you provided (2048x1024) looks in spectral space like this

shown is log10(abs(alm)) where alm is every spectral coefficient for degree l and order m. So the largest coefficient is O(1e26). We could just truncate to T341 even if higher resolution is used, but somehow the orography has a lot of power at higher wavenumbers. Any idea what we could do else? Problem is currently that the backtransform to grid-space creates NaNs...

[milankl]

Similar for the orography from the orog_TCO1279.nc file you shared, you can see where the overflow with Float32 kicks in

is there maybe something wrong with the spectral transform for high resolution, it looks so systematic

[milankl]

Okay we have a precision-issue with the spectral transform, this is not just the orography. I moved this into a new issue #104

[samhatfield]

Uh oh. Something going wrong with the summations in the Legendre transform?