Some datasets have deviating plev coordinates about esmvaltool_sample_data

[bouweandela]

So somewhere in the chain you want to make sure that all coordinates are consistent, right?

I agree, see my comments here ESMValGroup/ESMValCore#956 (comment)

[Peter9192]

Is this picked up by the cmor checks/fixes if the data is loaded in ESMValTtool?

[valeriupredoi]

no - why would it be? It's a valid plev coordinate wrt CMOR standards:

DimCoord(array([100000.00000001,  92500.00000001]), bounds=array([[103750.00000001,  96250.00000001],
       [ 96250.00000001,  88750.00000001]]), standard_name='air_pressure', units=Unit('Pa'), long_name='pressure', var_name='plev', attributes={'positive': 'down'})

is a perfectly fine coordinate 😃

[valeriupredoi]

note that this doesn't fix ESMValGroup/ESMValCore#956 - there is no need for a fix for that perse, it behaves as it should (I'll post a comment there); also this is not an issue 👍

[Peter9192]

Okay so maybe it's valid, but it's not a standard level. If you try to join two cubes in Iris where one of them has plev = [100000.00000001, 92500.00000001] and the other has plev=[100000., 92500.], it will fail because it cannot merge cubes with different coordinates. This makes a lot of sense to me.
So somewhere in the chain you want to make sure that all coordinates are consistent, right?

[valeriupredoi]

what do you mean they're not standard? ie they're not plev17 or plev19? That's true, but Z-data can be whatever the model outputs, as long as the units, standard name and order in which the points are (positive: up or down depending what type of coord is) are CMOR-standard all's fine. Note that iris' restriction applies to any difference in coordinate points values and attributes, no matter if they're CMOR-compliant or not, that's a self-imposed standard that the iris devs have implemented so that there are no unexpected issues, which in some cases, like here, is useful; in other cases we need to go around it and shunt it since it's a nuisance

[bouweandela]

Is this picked up by the cmor checks/fixes if the data is loaded in ESMValTtool?

Yes, this should issue a warning coming from here: https://github.com/ESMValGroup/ESMValCore/blob/b632f9ac7172a91b52fe91264f37a2ce3e13ac3e/esmvalcore/cmor/check.py#L633

[stefsmeets]

I made a PR that removes the deviating datasets from the repo, since we do not use the esmvalcore machinery to go through the fixes. See #21

[bouweandela]

I'm not entirely sure if this is a bad enough error for removing the datasets. If I understood it correctly, the plev coordinates are the same up to the numerical precision of the floating point number used to represent the coordinate data, right?

[Peter9192]

iris will say they are different. If these are still included, ESMValGroup/ESMValCore#968 will fail.

[bouweandela]

Yes, I think that is a usability problem of iris, because to a scientist this doesn't make sense, from a physical point of view they are the same. It would be good if the ESMValCore is able to deal with this kind of technical detail without bothering the scientist users with it.