Comments (13)
Indeed, neither the table nor the variable name depends on the experiment.
Additionally, some of these are actually cell_measures
, so the most elegant way, imho, to deal with them would be to add the fx files to the list of input filenames before they are loaded as cubes into iris.
Then, iris will automatically sort them out and make them available via cube.cell_measure()
.
Grepping the tables, this seems to apply for
areacella
from fxareacellg
from IfxAnt and IfxGreareacello
from Ofxvolcello
from either Ofx or (Omon, Oyr, Odec).areacellr
from fx
from esmvalcore.
Additionally, some of these are actually
cell_measures
, so the most elegant way, imho, to deal with them would be to add the fx files to the list of input filenames before they are loaded as cubes into iris.
Then, iris will automatically sort them out and make them available viacube.cell_measure()
.
Grepping the tables, this seems to apply for
Good call, I think this would also remove the UserWarning: Missing CF-netCDF measure variable ...
which appears very often (see #853).
from esmvalcore.
Another issue with fx file directory structure:
fx files are stored for each ensemble member and experiment individually, but should be independent of these keys (e.g., areacello is only dependent on grid structure, which is the same for each ensemble member). This causes unnecessary file duplication and the tool to crash for all ensemble members for which the fx files are not placed in their respective directories.
In CMIP5, fx files therefore were stored under r0i0p0. Maybe a similar solution could be implemented for CMIP6?
from esmvalcore.
cheers for the heads up @DSenftleben - I suggest we wait a bit until things get settled in terms of data storage structure and replication for CMIP6. Eg I am looking on the CEDA archive (BADC) and I am seeing only the amip
and the Efx dir is populated like this:
/badc/cmip6/data/CMIP6/CMIP/CNRM-CERFACS/CNRM-CM6-1/amip/r1i1p1f2/Efx/sftgrf/gr/latest/sftgrf_Efx_CNRM-CM6-1_amip_r1i1p1f2_gr.nc
whereas the fx
dir is populated as:
[valeriu@jasmin-sci2 ~]$ ls -la /badc/cmip6/data/CMIP6/CMIP/CNRM-CERFACS/CNRM-CM6-1/amip/r1i1p1f2/fx/
total 0
drwxr-x--- 1 badc open 0 Jan 24 17:01 .
drwxr-x--- 1 badc open 0 Jan 16 22:17 ..
drwxr-x--- 1 badc open 0 Jan 16 22:17 areacella
drwxr-x--- 1 badc open 0 Jan 24 17:00 areacellr
drwxr-x--- 1 badc open 0 Jan 16 22:17 mrsofc
drwxr-x--- 1 badc open 0 Jan 16 22:17 orog
drwxr-x--- 1 badc open 0 Jan 16 22:17 rootd
drwxr-x--- 1 badc open 0 Jan 16 22:17 sftgif
drwxr-x--- 1 badc open 0 Jan 16 22:17 sftlf
I'll ask the CEDA guys if this will change in the near future, if not, yes, we'll have to make the changes
from esmvalcore.
I noticed that the fx path settings are completely missing from config-developer.yml for CMIP6. Maybe someone can make a pull request to add those and that will also fix this issue?
from esmvalcore.
Some more context for the practicalities of fx files:
- Some "fx" files might be not so fxed: At least
thkcello
,masscello
, andvolcello
can live either inOfx
or in (Omon
andOdec
) depending on the model. My reading of the data request suggests that these are the only ones. - At least IPSL is embedding at least some fx information (certainly
areacello
) in their data files. Maybe we can just ignore that, but should inconsistencies surface, we should have a look there.
from esmvalcore.
Another issue with fx file directory structure:
fx files are stored for each ensemble member and experiment individually, but should be independent of these keys (e.g., areacello is only dependent on grid structure, which is the same for each ensemble member). This causes unnecessary file duplication and the tool to crash for all ensemble members for which the fx files are not placed in their respective directories.
In CMIP5, fx files therefore were stored under r0i0p0. Maybe a similar solution could be implemented for CMIP6?
a heads up that I have also noticed this for IPSL and CNRM-CERFACS: on BADC:
f1 = iris.load_cube('/badc/cmip6/data/CMIP6/CMIP/IPSL/IPSL-CM6A-LR/historical/r3i1p1f1/Ofx/areacello/gn/latest/areacello_Ofx_IPSL-CM6A-LR_historical_r3i1p1f1_gn.nc')
f2 = iris.load_cube('/badc/cmip6/data/CMIP6/CMIP/IPSL/IPSL-CM6A-LR/historical/r5i1p1f1/Ofx/areacello/gn/latest/areacello_Ofx_IPSL-CM6A-LR_historical_r5i1p1f1_gn.nc')
these files are totally identical (well, data is, almost all metadata is too, the parent ensemble is r1i1p1f1). Does any of you guys know what the strategy for including these files in the DRS is - @tillku is already starting to have headaches for his diagnostic that uses volcello
and that one is not yet present at all for any model on ESGF ๐บ
from esmvalcore.
OK guys I got confirmation of the standard CMIP6 fx paths from Ruth Petrie, data guru at STFC UK:
me: So, in a nutshell, should we expect to find fx files in a DRS like
/badc/cmip6/data/CMIP6/CMIP/[institute]/[dataset]/[experiment]/[ensemble]/[fxdir]/
where [fxdir] will be the same for each [ensemble] but have different names (fx, Ofx...) depending on [experiment]?
Ruth: Yes that is correct. It is formally โtable_idโ instead of โfx_dirโ the full list of all the table_ids is at https://github.com/WCRP-CMIP/CMIP6_CVs/blob/master/CMIP6_table_id.json this includes all the tables not just the fx ones but if you scan the list you should find all the different fx tables.
@zklaus : I think from the point of CMIP6 these fields are just normal variables.
Hence, they can be found as any other variable in the respective (Table,
variable) place.
Generally speaking, fixed files
for the ocean are in Ofx
, files for atmosphere and land are in fx
,
fixed files for the ice sheets over Antarctica and Greenland are in
IfxAnt
and IfxGre
, respectively, and some extended fixed things are
in Efx
.
from esmvalcore.
started working on a functional implementation in #1037 - let's take the discussion there, guys ๐บ
from esmvalcore.
#170 retrieves fx files as any other regular variable so when that gets merged we can close this one too ๐บ
from esmvalcore.
also maybe open a new issue with what @bouweandela has suggested to have a list of mip
's available for the fx data retriever to look for
from esmvalcore.
I think this is solved, isn't it? @valeriupredoi
from esmvalcore.
hell yeah, cheers for digging this up ๐บ
from esmvalcore.
Related Issues (20)
- Use iris' regridder caching for faster regridding? HOT 10
- concatenate experiments handles time ordering incorrectly HOT 5
- Weights for `climate_statistics` and `axis_statistics` are not lazy
- Preprocessor shape 'extract_shape' breaks if cell measure is time dependent
- [CI/GA] Tests fail with new issue (1 test) HOT 3
- Do we really need CMOR warnings? HOT 1
- Potential memory leak in local solar time preprocessor HOT 1
- [CI/CD/GA] Tests failing with one test fail HOT 4
- Using .json custom table HOT 15
- Multi-Model statistics failing due to different non-scalar aux coord
- Make `dask.yml` path configurable HOT 2
- `annual_statistics` may change dtype HOT 9
- New configuration file plan HOT 4
- Calendar conversion in the ETCCDI recipe
- Dataset wildcards may lead to "missing data" errors for derived variables
- Processing ERA5 (project native6) leads to huge dask graphs HOT 19
- Lots of iris warnings in output since iris v3.8.0 HOT 2
- Add `esmvaltool compare` command to run `esmvaltool/utils/testing/regression/compare.py` HOT 4
- GFDL-CM4 SfcWind height Issues HOT 1
- [testing/mamba] No useful output from mamba
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. ๐๐๐
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google โค๏ธ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from esmvalcore.