Apply cmor checks (and fixes) to fx-files about esmvalcore

[mattiarighi]

Some examples are already available in the fix files (like sftlf in EC-EARTH has wrong units and wrong latitude ordering).

Other models with wrong units in sftof are GFDL-CM2p1, GFDL-CM3, GFDL-ESM2M, and MIROC5.

[mattiarighi]

The units of sftlf and sftof are %, therefore valid min/max = 0/100 (as per cmor table).

Some models use a [0,1] range, which is also a fraction, but not in %, although the write % in the units attribute. Hence the units are correct, but the data are not consistent.

[mattiarighi]

Actually, they are CMOR variables, they have a dedicated table in CMIP5 and CMIP6.

It's not only about the 0-100, there is also a case (sftlf in EC-EARTH) where the latitude ordering is wrong and more errors might show up when we start using areacello and other fx-variables.

Since we already have a framework for fixing this kind of problems, we should use it.

[mattiarighi]

Can we now close this @jvegasbsc @valeriupredoi ?

[mattiarighi]

👍 = yes ?

[valeriupredoi]

hey @mattiarighi it would be very useful if you could provide us with corner cases that you have seen in which the fx mask files are weird (e.g. when the probability of land/ocean is from 0 to 1 instead of 0 to 100 etc). Cheers, man! 🍺

[valeriupredoi]

cool, cheers @mattiarighi -- I took a look at these ones you mention, and have already seen the 0->1 instead of 0->100, just to double check - there are no units for the sftlf/sftof fx variables (I mean, the units is % since it's a fraction, but it will always be % or could it be some crazy stuff like cats per square inch?)

[valeriupredoi]

got it, cheers!

[valeriupredoi]

hey @mattiarighi I have had a look through and thought about this issue and I think we shouldn't complicate the cmor checks with fx variable since, in effect, they are not real CMOR variables and the only bits that need to be fixed for fx files are grids and 0-100 percentages - my idea would be to create a module in preprocessor called _fix_auxiliary_files or smth like this and if get_fx_filelist() is called in _recipe.py whether it be for the preprocessor's mask_landea or for the variable[fx_files] or both, then run _fix_auxiliary_files.fix_fx() function that will handle the fixing, if needed, no matter what dataset/obs/obs4mips etc - these are small files and even if we need to load their data in memory it's still not much of a fuss. Guys @mattiarighi @bouweandela @jvegasbsc what say you?

[valeriupredoi]

damn! how have I not seen the CMIPX_fx Tables until now, nevermind, I just blabbered one comment up 🐄

[valeriupredoi]

👍 means 👍 just like Theresa may says 😆 I'll close this too

[valeriupredoi]

This is unfortunately not fully solved - see #1003 and ESMValGroup/ESMValTool#922 (comment) I will try fix it today

[valeriupredoi]

ok chaps so after much fiddling around with mostly dead-end options, I have come to propose a relatively straightforward method to address this issue: see example only for the derived vars that use fx files, extendable to all other fx cases picked up from recipe:

def _update_fx_settings(settings, variable, variables, config_user):
    """Find and set the FX derive/mask settings."""
    # update for derive
    if 'derive' in settings:
        fx_files = {}
        for var in get_required(variable['short_name']):
            if 'fx_files' in var:
                _augment(var, variable)
                fx_files.update(
                    get_input_fx_filelist(variable=var,
                                          rootpath=config_user['rootpath'],
                                          drs=config_user['drs'])
                )

                fx_vars = var['fx_files']
                fixed_fx_files = {}
############## TO PUT IN MODULE ###############################
                for fx_var in fx_vars:
                    fx_cube = iris.load_cube(fx_files[fx_var])
                    fix_metadata([fx_cube], fx_var, variable['project'],
                                 variable['dataset'], variable['cmor_table'],
                                 'fx', 'fx')
                    cmor_check_metadata(fx_cube, variable['cmor_table'],
                                        'fx', fx_var, 'fx')
                    iris.save(fx_cube, 'fx_some_cmor_filename.nc')
                    fixed_fx_files[fx_var] = 'fx_some_cmor_filename.nc'
######################## END OF MODULE #########################
        settings['derive']['fx_files'] = fixed_fx_files

so basically, right after the fx files are found (off whatever remote server or local DB) we fix and check them (the bits in comment anchors are to be converted into a preprocessing module that does that) and we save them in /preproc like any other CMIP file that was fixed, then update the fx_files[fx_var] dictionary with the pointers to the newly fixed and locally saved fx files and boom, the fixed files are being used rather than the shitty ones. Small acveat: cmor/check.py needs a conditional

        if self.frequency != 'fx':
            self._check_time_coord()
...
        if self.frequency != 'fx':
            self._add_auxiliar_time_coordinates()

to allow for fixes of cubes that don't have time coords (like the fx files that always have frequency = fx).
Off to pub now, will fix this proper on Monday and create a PRski 🍺

[valeriupredoi]

ESMValGroup/ESMValTool#1009

[zklaus]

So @valeriupredoi... does it work now? Can we build fixes for fx files?

[valeriupredoi]

it do, if you ask for an fx variable just like any other variable in #170

[zklaus]

I need it for a derived variable (#192), namely the fx variable volcello. Can I do that somehow?

[valeriupredoi]

fixed by #170 and #230