extra days calculated in some indices such as R10mm, R20mm about icclim

[armellereca]

One of our colleagues tested and compared cdos with icclim & below were the findings:

ESGF-Data is stored at 32bit floating point precision (~9 digits).

Our tests suggests, that CDO's will do calculations at a 64bit precision.
This leads to a more accurate result.
(e.g. if a mean value is calculated over a long period of time, the values can
add up to a large number, that might be problematic to handle at 32bit)
On the downside, going from 32 to 64bit can add noise to the data, as the precise values are not known.

ICCLIM uses the netCDF4 library. This library stores the data in a 32bit array.
At the moment, ICCLIM is not changing the dtype of the array and does
the calculations at 32bit.

This leads to rounding errors and especially if peak over threshold test are applied,
the results can differ.

Our suggestions:

1. Even through the data is stored at 32bit it makes sense to do calculations at 64bit.
2. As the data is only stored at 32bit it would be advisable to round the data before a peak over threshold test is applied. This would make sure that numeric noise does not affect the results.

[pagecp]

It is indeed a very good suggestion, it is much appreciated.

I will make the modifications to the code in this sense for a next icclim release.

Christian

On 14 Sep 2016, at 10:31, armellereca [email protected] wrote:

One of our colleagues tested and compared cdos with icclim & below were the findings:

ESGF-Data is stored at 32bit floating point precision (~9 digits).

Our tests suggests, that CDO's will do calculations at a 64bit precision.
This leads to a more accurate result.
(e.g. if a mean value is calculated over a long period of time, the values can
add up to a large number, that might be problematic to handle at 32bit)
On the downside, going from 32 to 64bit can add noise to the data, as the precise values are not known.

ICCLIM uses the netCDF4 library. This library stores the data in a 32bit array.
At the moment, ICCLIM is not changing the dtype of the array and does
the calculations at 32bit.

This leads to rounding errors and especially if peak over threshold test are applied,
the results can differ.

Our suggestions:

1. Even through the data is stored at 32bit it makes sense to do calculations at 64bit.
2. As the data is only stored at 32bit it would be advisable to round the data before a peak over threshold test is applied. This would make sure that numeric noise does not affect the results.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub #24 (comment), or mute the thread https://github.com/notifications/unsubscribe-auth/AG-dyvDULWRHIYLLfXIOGPMGkrT2fY06ks5qqCFFgaJpZM4J4DoL.

[MilkaRadojevic]

3. Floating is shown to have accurate precision for the counting-days indices since result is an integer number.
4. An issue that has been raised during clipc-icclim validation between cerfacs and cmcc: passing from 64bit to 32bit increases file size which in case of online portal and services should be avoided if not necessary.

[pagecp]

Only the calculations will be modified to be done in 64-bit, the results will still be stored as 32-bits.

On 11 Oct 2016, at 19:18, Milka Radojevic [email protected] wrote:

3. Floating is shown to have accurate precision for the counting-days indices since result is an integer number.
4. An issue that has been raised during clipc-icclim validation between cerfacs and cmcc: passing from 64bit to 32bit increases file size which in case of online portal and services should be avoided if not necessary.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub #24 (comment), or mute the thread https://github.com/notifications/unsubscribe-auth/AG-dyok6LPjXHalh4GXscGbc1N8WNdyUks5qy8T0gaJpZM4J4DoL.