Comments (18)
Some examples are already available in the fix files (like sftlf
in EC-EARTH
has wrong units and wrong latitude ordering).
Other models with wrong units in sftof
are GFDL-CM2p1
, GFDL-CM3
, GFDL-ESM2M
, and MIROC5
.
from esmvalcore.
The units of sftlf
and sftof
are %, therefore valid min/max = 0/100 (as per cmor table).
Some models use a [0,1] range, which is also a fraction, but not in %, although the write % in the units
attribute. Hence the units are correct, but the data are not consistent.
from esmvalcore.
Actually, they are CMOR variables, they have a dedicated table in CMIP5 and CMIP6.
It's not only about the 0-100, there is also a case (sftlf
in EC-EARTH
) where the latitude ordering is wrong and more errors might show up when we start using areacello
and other fx-variables.
Since we already have a framework for fixing this kind of problems, we should use it.
from esmvalcore.
Can we now close this @jvegasbsc @valeriupredoi ?
from esmvalcore.
👍 = yes ?
from esmvalcore.
hey @mattiarighi it would be very useful if you could provide us with corner cases that you have seen in which the fx mask files are weird (e.g. when the probability of land/ocean is from 0 to 1 instead of 0 to 100 etc). Cheers, man! 🍺
from esmvalcore.
cool, cheers @mattiarighi -- I took a look at these ones you mention, and have already seen the 0->1 instead of 0->100, just to double check - there are no units for the sftlf/sftof fx variables (I mean, the units is % since it's a fraction, but it will always be % or could it be some crazy stuff like cats per square inch?)
from esmvalcore.
got it, cheers!
from esmvalcore.
hey @mattiarighi I have had a look through and thought about this issue and I think we shouldn't complicate the cmor
checks with fx
variable since, in effect, they are not real CMOR variables and the only bits that need to be fixed for fx
files are grids and 0-100 percentages - my idea would be to create a module in preprocessor
called _fix_auxiliary_files
or smth like this and if get_fx_filelist()
is called in _recipe.py
whether it be for the preprocessor's mask_landea
or for the variable[fx_files]
or both, then run _fix_auxiliary_files.fix_fx()
function that will handle the fixing, if needed, no matter what dataset/obs/obs4mips etc - these are small files and even if we need to load their data in memory it's still not much of a fuss. Guys @mattiarighi @bouweandela @jvegasbsc what say you?
from esmvalcore.
damn! how have I not seen the CMIPX_fx Tables until now, nevermind, I just blabbered one comment up 🐄
from esmvalcore.
👍 means 👍 just like Theresa may says 😆 I'll close this too
from esmvalcore.
This is unfortunately not fully solved - see #1003 and ESMValGroup/ESMValTool#922 (comment) I will try fix it today
from esmvalcore.
ok chaps so after much fiddling around with mostly dead-end options, I have come to propose a relatively straightforward method to address this issue: see example only for the derived vars that use fx files, extendable to all other fx cases picked up from recipe:
def _update_fx_settings(settings, variable, variables, config_user):
"""Find and set the FX derive/mask settings."""
# update for derive
if 'derive' in settings:
fx_files = {}
for var in get_required(variable['short_name']):
if 'fx_files' in var:
_augment(var, variable)
fx_files.update(
get_input_fx_filelist(variable=var,
rootpath=config_user['rootpath'],
drs=config_user['drs'])
)
fx_vars = var['fx_files']
fixed_fx_files = {}
############## TO PUT IN MODULE ###############################
for fx_var in fx_vars:
fx_cube = iris.load_cube(fx_files[fx_var])
fix_metadata([fx_cube], fx_var, variable['project'],
variable['dataset'], variable['cmor_table'],
'fx', 'fx')
cmor_check_metadata(fx_cube, variable['cmor_table'],
'fx', fx_var, 'fx')
iris.save(fx_cube, 'fx_some_cmor_filename.nc')
fixed_fx_files[fx_var] = 'fx_some_cmor_filename.nc'
######################## END OF MODULE #########################
settings['derive']['fx_files'] = fixed_fx_files
so basically, right after the fx files are found (off whatever remote server or local DB) we fix and check them (the bits in comment anchors are to be converted into a preprocessing module that does that) and we save them in /preproc
like any other CMIP file that was fixed, then update the fx_files[fx_var]
dictionary with the pointers to the newly fixed and locally saved fx files and boom, the fixed files are being used rather than the shitty ones. Small acveat: cmor/check.py
needs a conditional
if self.frequency != 'fx':
self._check_time_coord()
...
if self.frequency != 'fx':
self._add_auxiliar_time_coordinates()
to allow for fixes of cubes that don't have time coords (like the fx files that always have frequency = fx).
Off to pub now, will fix this proper on Monday and create a PRski 🍺
from esmvalcore.
from esmvalcore.
So @valeriupredoi... does it work now? Can we build fixes for fx files?
from esmvalcore.
it do, if you ask for an fx variable just like any other variable in #170
from esmvalcore.
I need it for a derived variable (#192), namely the fx variable volcello. Can I do that somehow?
from esmvalcore.
from esmvalcore.
Related Issues (20)
- Add `esmvaltool compare` command to run `esmvaltool/utils/testing/regression/compare.py` HOT 4
- GFDL-CM4 SfcWind height Issues HOT 1
- [testing/mamba] No useful output from mamba
- Improve the ESMValCore release documentation
- Automatically select a lazy regridding scheme in `regrid` preprocessor function for 2D lat/2D lon grids when possible HOT 6
- Drop support for Python 3.9 HOT 3
- Update Iris pin to avoid using versions with memory issues HOT 4
- Increase the version number for 2.11.0 release
- Add draft release notes for v2.11.0rc1 HOT 1
- Remove `compilers` from `environment.yml`
- Recipe test results for ESMValCore v2.11.0rc1 HOT 15
- Diagnostic failure for `recipe_martin18grl.yml` on `v2.11.0rc1` HOT 1
- Diagnostic failure for `recipe_pcrglobwb.yml` on `v2.11.0rc1` HOT 1
- Diagnostic failure for `recipe_lauer22jclim_fig3-4_zonal.yml` on `v2.11.0rc1` HOT 1
- Add native6, OBS6 and RAWOBS rootpaths to metoffice config-user.yml template, and remove temporary dir
- Add control over chunks at load time
- Interpolation in `resample_hours`
- Adding ETCCDI custom CMOR tables for HadEX3 CMORizer and IPCC diagnotics HOT 1
- Make extract_point/extract_location/extract_coordinate_points preprocessor functions lazy HOT 1
- Lazy derive preprocessor function
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from esmvalcore.