Missing frequency using CMIP6 and obs4mips datasets about esmvalcore

[tomaslovato]

🔝 I'll do it !!

[tomaslovato]

@bouweandela I tested again a recipe with chl data from CMIP6 models and it worked fine.

[tomaslovato]

The problem here is the assumption that table.frequency is defined also in the header section of CMIP6 tables, but it was changed in favor of having frequency defined for each variable.

The solution is to modify ESMValTool/esmvaltool/cmor/table.py as follow

@@ -102,15 +102,16 @@ class CMIP6Info(object):
             self.tables[table.name] = table

             generic_levels = header['generic_levels'].split()
-            table.frequency = header.get('frequency', '')
             table.realm = header.get('realm', '')

             for var_name, var_data in raw_data['variable_entry'].items():
                 var = VariableInfo('CMIP6', var_name)
                 if 'frequency' in var_data:
                     var.frequency = var_data['frequency']
+                    table.frequency = var.frequency
                 else:
-                    var.frequency = table.frequency
+                    raise ValueError('missing frequency for var ' +
+                        var_name + ' in ' + header['table_id'])
                 var.read_json(var_data)
                 self._assign_dimensions(var, generic_levels)
                 table[var_name] = var

I tested it and it only requires the removal of ESMValTool/esmvaltool/cmor/tables/cmip6/Tables/CMIP6_CFsubhrOff.json that is obsolete and no more included in CMOR tables (I missed that in the previous update PR)

I think it should be ok coded in this way :

• still define table.frequency based on variables (by construction they have all the same freq within a table) in order to be compatible with CMIP5 table definition and its use in code workflow.
• a safety control on frequency definition for each variable

@valeriupredoi @mattiarighi Any suggestion on how to implement the bugfix?

[mattiarighi]

This seems to affect obs4mips data as well. I get exactly the same error with the GPCP-SG and CERES-EBAF datasets.

[valeriupredoi]

for chl in CMIP6_Omon.json

        "chl": {
            "frequency": "mon",

we can't hack CMOR tables, if they are obsolete then we update those via git rather than hacking them manually.
Can you provide a minimal recipe that I can use to reproduce this issue pls? 🍺

[valeriupredoi]

ah no need to worry about a recipe example, it fails straight away for tas as well - question is - has the current implementation worked at__all for CMIP6 data?

[valeriupredoi]

it works if you add it in the recipe: frequency: mon, so worst case scenario we add it in recipe than we change the CMOR tables, but am looking at a more respectable change at the mo 😁

[tomaslovato]

for chl in CMIP6_Omon.json

        "chl": {
            "frequency": "mon",

we can't hack CMOR tables, if they are obsolete then we update those via git rather than hacking them manually.
Can you provide a minimal recipe that I can use to reproduce this issue pls? 🍺

@valeriupredoi My bad!! I forgot to say that with the correction proposed there is no need to hack cmor tables.

[schlunma]

ah no need to worry about a recipe example, it fails straight away for tas as well - question is - has the current implementation worked at__all for CMIP6 data?

Yes, ~ one week ago I successfully ran this:

https://github.com/ESMValGroup/ESMValTool/blob/864260c8e27f15d8a34b08be61372e8a3db32432/esmvaltool/recipes/recipe_ecs.yml#L77-L95

[valeriupredoi]

OK here's the bugger: in table.py: table.frequency = header.get('frequency', '') -- problem is in the latest CMIP6 tables the Header member does not contain the frequency anymore but it is specified at variable levels; also the same variable shows up in different modeling realms and the realm is treated as a single concatenated string; I changed it a bit like this:

    def _load_table(self, json_file):
        with open(json_file) as inf:
            raw_data = json.loads(inf.read())
            if not self._is_table(raw_data):
                return
            table = TableInfo()
            header = raw_data['Header']
            table.name = header['table_id'][6:].split('_')[-1]
            self.tables[table.name] = table
            generic_levels = header['generic_levels'].split()
            table.realm = header.get('realm', '').split()

            for var_name, var_data in raw_data['variable_entry'].items():
                for realm in table.realm:
                    if var_data['modeling_realm'] == realm:
                        var = VariableInfo('CMIP6', var_name)
                        if 'frequency' in var_data:
                            frequency = var_data['frequency']
                        else:
                            frequency = header.get('frequency', '')
                        var.read_json(var_data)
                        var.frequency = frequency
                        var.modeling_realm = realm
                        self._assign_dimensions(var, generic_levels)
                        table[var_name] = var

and I checked that the table instance returned does indeed contain a valid frequency, but I still get the checker fail, the problem should be spread to the checker as well....I'll look more into it tomorrow

[tomaslovato]

@valeriupredoi Why not to try to simply force assignment of frequency to the table as I did in the solution proposed above. Here below a simple addition to your code ...

                    if var_data['modeling_realm'] == realm:
                        var = VariableInfo('CMIP6', var_name)
                        if 'frequency' in var_data:
                            frequency = var_data['frequency']
                        else:
                            frequency = header.get('frequency', '')
                        var.read_json(var_data)
                        var.frequency = frequency
                        var.modeling_realm = realm
+                       table.frequency = frequency

As part of previous consideration, this assignment should be kept to avoid compatibility issue with CMIP5 tables.

The obsolete table I was referring (ESMValTool/esmvaltool/cmor/tables/cmip6/Tables/CMIP6_CFsubhrOff.json) was removed from the CMOR repository and it is a leftover from PR #937 that need to be removed as well.

[valeriupredoi]

hey @tomaslovato and @mattiarighi have a look at this one ESMValGroup/ESMValTool#985 and test please, when you gots time, I will fix the test as soon as I hear you guys had success testing!
@tomaslovato cheers for the suggestion - have a look at the code changes in the PR pls, it is pretty much just as you suggest, only that the table gets forced a frequency attribute depending on var and mip right before the table is passed over to the checks

[tomaslovato]

hey @tomaslovato and @mattiarighi have a look at this one #985 and test please, when you gots time, I will fix the test as soon as I hear you guys had success testing!
@tomaslovato cheers for the suggestion - have a look at the code changes in the PR pls, it is pretty much just as you suggest, only that the table gets forced a frequency attribute depending on var and mip right before the table is passed over to the checks

[mattiarighi]

But why is this affecting obs4mips as well?
We didn't change those tables, did we?

[valeriupredoi]

Good question - it's the checker that is calling CMIPXInfo (there is no function dedicated to obs4mips info from CMOR Table) and the problem was there - in fact, I am not even 100% sure why we even had the problem after all, for any model, since the syntax in CMIP6Info() was correct and accounting for the two cases that frequency can be in the var dictionary or the header of the CMOR Table. Forcing the frequency is a kludge, we'll have to figure it out why was missing in the first place (well, there were all sorts of problems with table.py, but even after fixing them that bloody freq was on and off) Dr Valeriu Predoi. Computational scientist NCAS-CMS University of Reading Department of Meteorology Reading RG6 6BB United Kingdom

…

On Wed, 20 Mar 2019, 07:38 Mattia Righi, ***@***.***> wrote: But why is this affecting obs4mips as well? We didn't change those tables, did we? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <https://github.com/ESMValGroup/ESMValTool/issues/981#issuecomment-474719899>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AbpCo6bMFhiLE2ax-25fmTeIwRq3xO3Wks5vYeWKgaJpZM4b6V3x> .

[valeriupredoi]

dereferencing ESMValTool closed PR #985 and referencing #158

[bouweandela]

@tomaslovato It looks like this issue may have been solved. Can you confirm this?