cerfacs-globc / icclim

icclim: Python library for climate indices and climate indicators calculation.

icclim is a Python library to compute climate indices. icclim name stands for index, calculation, climate.

Installation

From pypi: pip install icclim.

From conda-forge: conda install -c conda-forge icclim.

From sources:

• Clone the repository git clone https://github.com/cerfacs-globc/icclim.git
• Install icclim pip install .

How to use icclim

Let's count the number of days above 25°C for each year, which corresponds to the index SU, from a tasmax variable scattered in multiple netcdf files.

SU is one of the many index that can be computed with icclim. See the documentation to explore what other index you can compute with icclim.

import icclim

summer_days = icclim.su(
    "netcdf_files/tasmax_1990-2100.nc", out_file="summer_days.nc"
)

For more examples on how to use icclim, see icclim's How to ... documentation or our notebooks.

Who use icclim

icclim is part of C4I platform backend and is integrated in CLIPC Portal. icclim is also used by some independent researchers.

Who made icclim

icclim has always been an open source project and was successfully made thanks to the joint effort of all its contributors. The lead development is made at CERFACS, a research institution located in Toulouse, France.

Grants

This open-source project has been possible thanks to funding by the European Commission projects:

• FP7-CLIPC (2013-2016)
• FP7-IS-ENES2 (2013-2017)
• EUDAT2020 (2015-2018)
• H2020-IS-ENES3 (2019-2023)

The beautiful icclim logo is a creation of Carole Petetin and has been funded by the H2020 IS-ENES3 project grant agreement No 824084 (2019-2023).

Indices

For a detailed description of each ECA&D index, please visit: https://www.ecad.eu/documents/atbd.pdf

icclim's Issues

extra days calculated in some indices such as R10mm, R20mm

Dear icclim gurus,

we tried to compare indices derived from CDOs and from icclim. the results were pretty similar. however, there is some discrepancies such as rounding of values in the arrays.
for example in R10mm, the value of 9.999936 is treated as 10 in icclim, hence one day is added to the icclim-calculated index. so first question, has anyone have similar experience? with discussion from a python expert here, it could be the dtype of icclim arrays are set to float32. i will try to check this and/or change the arrays to float64 for similar indices with thresholds.

second question, digging deeper to the calculation of precipitation, sometimes I see the warning of units of "arr" must be in mm/day. however our input from the ESGF is always in kg/s. Should we ignore these warnings?

thanks!

def R10mm_calculation(arr, fill_val=None, threshold=10.0, out_unit="days"):
'''
Calculates the R10mm indice: number of heavy precipitation days (i.e. days with daily precipitation amount > = 10 mm) [days]

:param arr: daily precipitation (liquid form) flux (e.g. "pr") in mm/day
:type arr: numpy.ndarray (3D) or numpy.ma.MaskedArray (3D)
:param fill_val: fill value 
:type fill_val: float

:rtype: numpy.ndarray (2D)        (if "arr" is numpy.ndarray)
     or numpy.ma.MaskedArray (2D) (if "arr" is numpy.ma.MaskedArray)

.. warning:: Units of "arr" must be mm/day.

.. warning:: If "arr" is a masked array, the parameter "fill_val" is ignored, because it has no sense in this case.
'''


R10mm = calc.get_nb_events(arr, logical_operation='get', thresh=threshold, fill_val=fill_val, out_unit=out_unit)

return R10mm

Performance of existing bootstrap algorithm

A discrepancy in the performance when running the bootstrap algorithm at has been reported and it would be good to pin down the exact cause. If it turns out to be related to the required amount of memory it would be good to get a rough idea how ICCLIM handles memory, e.g. if it simply scales with input file size.

NetCDF files are opened and close several times

At the beginning of icclim, the netcdf data are opened and close in many different functions. It is required to "stream" the nc file across the different functions and open it one only time.

NumPy print option threshold error

@basmac reported on the ocgis tracker an issue (NCPP/ocgis#499) with print thresholds:

File "/usr/local/gfortran-8.3/anaconda3/lib/python3.6/site-packages/icclim/calc_percentiles.py", line 10, in 
numpy.set_printoptions(threshold=numpy.nan)
File "/usr/local/gfortran-8.3/anaconda3/lib/python3.6/site-packages/numpy/core/arrayprint.py", line 246, in set_printoptions
floatmode, legacy)
File "/usr/local/gfortran-8.3/anaconda3/lib/python3.6/site-packages/numpy/core/arrayprint.py", line 93, in _make_options_dict
raise ValueError("threshold must be numeric and non-NAN, try "
ValueError: threshold must be numeric and non-NAN, try sys.maxsize for untruncated representation

This may be resolved in newer icclim versions.

360_day calendar tests

360_day branch is technically working but further test to verify the implementation independently are needed (as discussed). Could we list the recommended short lists of indices to include for this testing in this issue?

5.0.0 roadmap

We are very close to releasing a 5.0.0 version of icclim but a few points should be considered before the release.
These are suggestions not an exact roadmap to follow.

Clean 4.x.x opened issues

We have a few (10) trailing issues from 4.x.x versions of icclim.
We should consider reviewing each one of them and close those which are now irrelevant.
I suggest to keep feature request if there are any.
For bugs, we should probably redirect users to the 5.0.0 version (once it is released).

Improve documentation

• We should add an article explaining why icclim 5.0.0 exist now and how it differs from xclim.: #107
• We might want to add an article comparing the results of icclim and climpact, make sure we don't have a license issue with that because climpact is in GPL 3.
• Add dev documentation articles to explain:
  • how to contribute to icclim
  • how to release icclim
  • what does our CI do (it is split between github actions, pre-commit ci and readthedoc PR hook)
• We should consider adding a PR template with a check list (xclim has a good example of this)
• We should consider adding an issue template to help user describe their issue (and ease their reading for ourselves)
• Update tags on github with "xclim", "xarray", "dask"
• Add a "how to" article for the Dask usage.
• Add a comprehensive table of the precise and exact definitions of each climate index. I (Christian) thought this has always been missing, even though there is the pdf available for example for ECA&D. This will also help to be properly referenced in google for users to find the package.
• Bring examples up-to-date with new version (for example "index TG90p with callback" may not be relevant anymore).
• Reference the notebooks developed for C4I.
• have a Gallery of a few examples using those notebooks.

Run a campaign on windows

All dev have been made on unix like environment and we already have a bug on windows #108 .
It would be good to run a few tests on windows before releasing.

Publish to conda forge

We already have requests to publish icclim to conda forge.

A pretty logo

It would be nice to have a logo for icclim.
It would make sense to display it on readthedoc and on the presentations we do.

Unit tests

For now icclim has very few unit tests.
We should consider setting a minimal coverage goal and display the actual coverage on pull requests and our github home page.

ping: @pagecp

ICCLIM does not work with python netcdf library ^1.4.0

Hi,

It seems that icclim does not work with the latest python netcdf library. I was able to get it working with version 1.3.1 (pip install netcdf4==1.3.1)

It seems that the line

import netcdftime

does not work with higher python netcdf4 versions than 1.3.1

Cheers,
Maarten

Bug in get_percentile_arr function

The get_percentile_arr function in calc_percentiles attempts to access the .mask attribute of the data array. But the mask attribute does not always return a mask array (see masked array documentation). Its more reliable to use the numpy.ma.getmaskarray(..) function.

where is indice_group?

Is this stored somewhere? It is the one parameter I could not track down!

https://github.com/tatarinova/icclim/blob/test/icclim/set_globattr.py#L4

For EUR-11 missing vars: x,y (as in rlon,rlat on EUR-44), rotated_pole and glob attributes

EUR-11 ensemble indices files are missing:

1. The variables x,y (analogue to rlon,rlat on EUR-44)
2. Variable rotated_pole

And the attribute grid_mapping: "rotated pole"

Seems this causes issues in adaguc visualization and quicklooks on climat4impact portal.

See 3 ensemble netcdf headers comparison here (from left to right 2deg, 0p5deg and 0p11deg):

Refactor to-do list

A to-do list for ICCLIM refactoring (=cosmetic changes that does not change the output of the code).

• Restyle the code along the lines recommended in the PEP8 standard
• In calc_ind.py, function zzz, use the locals()-construct such that the if-statements can be removed.

_FillValue turned to zero non-missing for the indices based on counting threshold e.g. WSDI

In grid points with _FillValue, WSDI is set to zero instead of _FillValue.

Example 3 does not run with Python 3

Example 3: indice TG90p with callback
runs it with py2.7 and icclim V4.2.11
crashes with py3.6 and icclim V4.2.13
i get:

2019-12-11 10:23:09,491    ********************************************************************************************
2019-12-11 10:23:09,491    *                                                                                          *
2019-12-11 10:23:09,493    *          icclim                                                            V4.2.13       *
2019-12-11 10:23:09,493    *                                                                                          *
2019-12-11 10:23:09,493    *                                                                                          *
2019-12-11 10:23:09,493    *          Wed Dec 11 10:23:09 2019 GMT                                                    *
2019-12-11 10:23:09,493    *                                                                                          *
2019-12-11 10:23:09,493    *          BEGIN EXECUTION                                                                 *
2019-12-11 10:23:09,493    *                                                                                          *
2019-12-11 10:23:09,493    ********************************************************************************************
2019-12-11 10:24:45,325 Loading data...
2019-12-11 10:24:57,910 Daily Percentiles calculation for out-of-base years. Please be patient...
Traceback (most recent call last):
  File "test_TG90p.py", line 32, in <module>
    icclim.indice(indice_name='TG90p', in_files=filenames, var_name='tas', slice_mode='AMJJAS', time_range=[dt1, dt2], base_period_time_range=[base_dt1, base_dt2], out_file=out_f, out_unit='%', callback=callback.defaultCallback2)
  File "/work/ch0636/g300046/conda_envs/py3/lib/python3.6/site-packages/icclim-4.2.13-py3.6.egg/icclim/icclim.py", line 523, in indice
    save_percentile=save_percentile) ## tuple: (dt_centroid_arr, dt_bounds_arr, indice_arr) 
  File "/work/ch0636/g300046/conda_envs/py3/lib/python3.6/site-packages/icclim-4.2.13-py3.6.egg/icclim/icclim.py", line 962, in get_indice_from_dict_temporal_slices
    bootstrapping=bootstrapping)
  File "/work/ch0636/g300046/conda_envs/py3/lib/python3.6/site-packages/icclim-4.2.13-py3.6.egg/icclim/calc_percentiles.py", line 382, in get_percentile_dict
    percentile_array = return_perc_array_2_compute_bootstrapping(dt_arr, arr_filled, t_units, t_calendar, ytd, window_width, percentile, interpolation, ignore_Feb29th, only_leap_years, bootstrapping)
  File "/work/ch0636/g300046/conda_envs/py3/lib/python3.6/site-packages/icclim-4.2.13-py3.6.egg/icclim/calc_percentiles.py", line 298, in return_perc_array_2_compute_bootstrapping
    percentile_array[i,:,:] = np.percentile(arr_filled[ind_2_calc,:,:], percentile, axis=0, interpolation=interpolation)
IndexError: index 3652 is out of bounds for axis 0 with size 3652

Drop python 3.7

• icclim version: 5.0.0rc2

• Python version: 3.7

Description

Xclim (and others) are dropping support for python 3.7 Ouranosinc/xclim#966 as it will only receive security fixes.
We should follow the movement and drop 3.7 as well.

Documentation -- netCDF4 version 1.2.9 required

With netCDF4 version 1.2.7 icclim crashed for certain files:
File "netCDF4/_netCDF4.pyx", line 4116, in netCDF4._netCDF4.Variable.setitem (netCDF4/_netCDF4.c:47421)
IndexError: in the future, 0-d boolean arrays will be interpreted as a valid boolean index

I was not able to pinpoint the exact cause of this error (neither in the file structure nor in the code). But the problem disappeared after upgrading to netCDF4 version 1.2.9. (see [netcdf4-python issue #521] (Unidata/netcdf4-python#521)

It would be appropriate to state this version requirement in the dependencies section of the readthedocs documentation.

[xarray] Rx5day indice computation with xarray_new_version

Rx5day must be implemented to work on xarray_new_version branch

Test case:

1. Retrieve a dataset with pr variable on c4i
2. Compute rx5day indice with icclim locally
3. Use a tool such a Panoply to ensure the rx5day computed is correct

NameError: name 'config_structure' is not defined

With the latest modifications, the following error occurs when trying to import:
NameError: name 'config_structure' is not defined

[icclim4.1.0] KeyError: 'files_years'

When trying to run icclim4.1.0 the error message appears at line 404 in the icclim.py" (see below).
Note that icclim4.0.3 does not show this type of error.

It looks like an issue with defining time array. Any thoughts on this? Solution?

[milka@nemoglobc1 icclim-4.1.0]$
CLIMATE INDEX: TX10p
INPUT VARNAME: tasmax
INPUT FILE: /scratch/globc/milka/CLIPC/icclim-4.1.0/cat_tasmax_1d_EUR-11_19890101-20101231_EURO4M_MESANv1.nc
1989-01-01 00:00:00 2010-12-31 00:00:00 1989-01-01 00:00:00 2010-12-31 00:00:00
Loading data: chunk 1/16 ...
Loading data: chunk 2/16 ...
Traceback (most recent call last):
File "./run_icclim4.py", line 116, in
icclim.indice(indice_name=Indice_Name, in_files=File_InVar, var_name=Name_InVar, time_range=[dt1, dt2], base_period_time_range=[ref_dt1, ref_dt2], out_unit='days', out_file=Indice_OutFile, transfer_limit_Mbytes=500, callback=callback.defaultCallback2)
File "/scratch/globc/milka/CLIPC/icclim-4.1.0/icclim/icclim.py", line 404, in indice
nc = MFDataset(VARS[v]['files_years'].keys(), 'r', aggdim=dim_name) # VARS[v]['files_years'].keys(): files of current variable
KeyError: 'files_years'

icclim doesn't support NetCDF4/HDF5 from CMIP6 as reported

Does anyone has an exprience with running icclim on the new set of CMIP6 simulations?
Laurent Terray (Cerfacs) runs them on the CEDA's jasmin server. It seams more as a compatibility issue in compilation with netcdf4/hdf5 lib.

Any thoughts on this?

Many thanks, Milka
ps
Yes, I am back :)

out of bounds for time coordinates

RX1Day using icclim-4.2.4: UnboundLocalError: local variable 'intersecting_years' referenced before assignment

Hallo, I tried to calculate the RX1DAY for evaluation period of REMO and CLM.

python script:

indices = ['RX1DAY']
mode_of_slices = ['year']
models = ['MPI-CSC-REMO2009', 'CLMcom-CCLM4-8-17']
sim='EUR44'
inputdir=BASE+sim+'/daily/download/'+var+'/'
outdir='/work/m300021/highendex/indices/outclim/'

dt1 = datetime.datetime(1989,1,2)
dt2 = datetime.datetime(2008,12,31)

for index in indices:
for model in models:
for mode_of_slice in mode_of_slices:
FILES = sorted(get_files(inputdir,'ERAINT'+model+'*'))
print(index,FILES)
os.chdir(inputdir)
out_f = outdir+index+''+mode_of_slice+''+sim+'ECMWF-ERAINT_evaluation_r1i1p1'+model+'_v1_day_1989-2008.nc'
icclim.indice(indice_name=index, in_files=FILES, var_name=var, slice_mode=mode_of_slice,
time_range=[dt1, dt2], out_file=out_f)

python output:

error: ('RX1DAY', ['pr_EUR-44_ECMWF-ERAINT_evaluation_r1i1p1_MPI-CSC-REMO2009_v1_day_19890102-19901231.nc', 'pr_EUR-44_ECMWF-ERAINT_evaluation_r1i1p1_MPI-CSC-REMO2009_v1_day_19910101-19951231.nc', 'pr_EUR-44_ECMWF-ERAINT_evaluation_r1i1p1_MPI-CSC-REMO2009_v1_day_19960101-20001231.nc', 'pr_EUR-44_ECMWF-ERAINT_evaluation_r1i1p1_MPI-CSC-REMO2009_v1_day_20010101-20051231.nc', 'pr_EUR-44_ECMWF-ERAINT_evaluation_r1i1p1_MPI-CSC-REMO2009_v1_day_20060101-20081231.nc'])

UnboundLocalError Traceback (most recent call last)
in ()
16 out_f = outdir+index+''+mode_of_slice+''+sim+'ECMWF-ERAINT_evaluation_r1i1p1'+model+'_v1_day_1989-2008.nc'
17 icclim.indice(indice_name=index, in_files=FILES, var_name=var, slice_mode=mode_of_slice,
---> 18 time_range=[dt1, dt2], out_file=out_f)

icclim-4.2.4-py2.7.egg/icclim/icclim.pyc in indice(in_files, var_name, indice_name, slice_mode, time_range, out_file, threshold, N_lev, transfer_limit_Mbytes, callback, callback_percentage_start_value, callback_percentage_total, base_period_time_range, window_width, only_leap_years, ignore_Feb29th, interpolation, out_unit, netcdf_version, user_indice)
553 ignore_Feb29th=ignore_Feb29th, interpolation=interpolation,
554 out_unit=out_unit,
--> 555 user_indice=user_indice) ## tuple: (dt_centroid_arr, dt_bounds_arr, indice_arr)
556 indice_arr_current_chunk = indice_tuple_current_chunk[2]
557

icclim-4.2.4-py2.7.egg/icclim/icclim.pyc in get_indice_from_dict_temporal_slices(indice_name, vars_dict, thresh, window_width, only_leap_years, callback, callback_percentage_start_value, callback_percentage_total, ignore_Feb29th, interpolation, out_unit, user_indice)
909
910 ### for out-of-base years
--> 911 if slice[1] not in intersecting_years: # slice[1] --> year
912 current_intersecting_year = -9999
913 reduced_base_years_list = [-9999]

UnboundLocalError: local variable 'intersecting_years' referenced before assignment

Parallel implementation of bootstrap algorithm

Missing _FillValue should be handled nicely

Missing _FillValue attribute of the input variable in an input file results in a crash. This is not desirable as _FillValue is not a required attribute according to NUG and CF.
A possible solution could as follows:
In icclim.py, near line 245, change code to read

try: ####----- LB 20170818
    fill_val = ncVar._FillValue.astype('float32') # fill_value must be the same type as "ind_type", i.e. 'float32'
except AttributeError: ####----- LB 20170818
    fill_val = numpy.float32(1.e20) # just an example literal constant ####----- LB 20170818

and similar changes near line 410. This seems to work for some test files, but I leave it to experts to check more carefully, then decide and implement suitable code.

RuntimeError: NetCDF: Not a valid data type or _FillValue type mismatch

Hi dear,
I've got the following error when I want to run the code in the below:

import icclim
from icclim.util import callback
my_indice_params = {'indice_name': 'my_indice', 'calc_operation': 'nb_events', 'logical_operation': 'lt', 'thresh': 0 + 273.15 } file_tas = "/home/2002-T2max.nc"
out_f = "my_indice.nc"
icclim.indice(user_indice=my_indice_params, in_files=file_tas, var_name='tas', slice_mode='year', out_unit='days', out_file=out_f, callback=callback.defaultCallback2)

2021-09-14 15:26:48,744 ********************************************************************************************
2021-09-14 15:26:48,744 * *
2021-09-14 15:26:48,745 * icclim V4.2.19 *
2021-09-14 15:26:48,745 * *
2021-09-14 15:26:48,745 * *
2021-09-14 15:26:48,745 * Tue Sep 14 15:26:48 2021 GMT *
2021-09-14 15:26:48,745 * *
2021-09-14 15:26:48,745 * BEGIN EXECUTION *
2021-09-14 15:26:48,745 * *
2021-09-14 15:26:48,745 ********************************************************************************************
Traceback (most recent call last):
File "", line 1, in
File "/home/icclim/icclim/icclim.py", line 175, in indice
indice_dim = util_nc.copy_var_dim(inc, onc, var_name[0], lev_dim_pos=lev_dim_pos) # tuple ('time', 'lat', 'lon')
File "/home/icclim/icclim/util/util_nc.py", line 233, in copy_var_dim
onc_dim0 = onc.createVariable( str(time_var), inc_dim0.dtype, (str(time_var)) ) # time
File "src/netCDF4/_netCDF4.pyx", line 2748, in netCDF4._netCDF4.Dataset.createVariable
File "src/netCDF4/_netCDF4.pyx", line 3808, in netCDF4._netCDF4.Variable.init
File "src/netCDF4/_netCDF4.pyx", line 1927, in netCDF4._netCDF4._ensure_nc_success
RuntimeError: NetCDF: Not a valid data type or _FillValue type mismatch

Also ncdump result is as follow:

$ ncdump -h ../2002-T2max.nc

netcdf \2002-T2max {
dimensions:
time = 365 ;
lat = 121 ;
lon = 121 ;
variables:
int64 time(time) ;
time:long_name = "time" ;
time:standard_name = "time" ;
time:axis = "T" ;
time:stored_direction = "increasing" ;
time:type = "double" ;
time:units = "days since 2002-01-01 00:00:00" ;
time:calendar = "proleptic_gregorian" ;
int64 realization ;
realization:long_name = "realization" ;
realization:units = "1" ;
realization:standard_name = "realization" ;
realization:stored_direction = "increasing" ;
realization:type = "integer" ;
double lat(lat) ;
lat:_FillValue = NaN ;
lat:units = "degrees_north" ;
lat:standard_name = "latitude" ;
lat:long_name = "latitude" ;
lat:stored_direction = "decreasing" ;
lat:axis = "Y" ;
lat:positive = "up" ;
lat:type = "double" ;
lat:valid_max = 90. ;
lat:valid_min = -90. ;
double lon(lon) ;
lon:_FillValue = NaN ;
lon:units = "degrees_east" ;
lon:standard_name = "longitude" ;
lon:long_name = "longitude" ;
lon:axis = "X" ;
lon:positive = "up" ;
lon:type = "double" ;
lon:valid_max = 360. ;
lon:valid_min = -180. ;
float tas(time, lat, lon) ;
tas:_FillValue = NaNf ;
tas:long_name = "Near-Surface Air Temperature" ;
tas:units = "K" ;
tas:standard_name = "air_temperature" ;
tas:comment = "near-surface (usually, 2 meter) air temperature" ;
tas:cds_magics_style_name = "near-surface-air-temperature" ;
tas:type = "real" ;
tas:coordinates = "realization" ;

// global attributes:
:Conventions = "CF-1.7" ;
:institution = "European Centre for Medium-Range Weather Forecasts" ;
:history = "2021-09-13T10:14 GRIB to CDM+CF via cfgrib-0.9.9.0/ecCodes-2.16.0 with {"source": "/cache/data0/e27cfa3c612c16349f2e2306e1fbc570.grib", "filter_by_keys": {}, "encode_cf": ["parameter", "time", "geography", "vertical"]}" ;
:source = "ECMWF" ;
}

Also my netcdf file has been attached.

2002-T2max.zip

Cheers,

FillValue attribute error

Running icclim using python3.7 with the latest netcd4 and cftime packages installed produces the following:

    icclim.indice(indice_name='RX5day', in_files=files, var_name='RAINTOT', time_range=[dt1, dt2], slice_mode='month', out_file=out_f)
  File "icclim.py", line 216, in indice
    indice_dim = util_nc.copy_var_dim(inc, onc, var_name[0], lev_dim_pos=lev_dim_pos) # tuple ('time', 'lat', 'lon')    
  File "icclim/util/util_nc.py", line 274, in copy_var_dim
    onc_dim0.__setattr__(attr[0], attr[1])
  File "netCDF4/_netCDF4.pyx", line 4311, in netCDF4._netCDF4.Variable.__setattr__
AttributeError: _FillValue attribute must be set when variable is created (using fill_value keyword to createVariable)

The accompanying ncdump from the file that we are using

netcdf daily_RAINTOT_d01 {
dimensions:
	time = UNLIMITED ; // (31 currently)
	lat = 152 ;
	lon = 223 ;
variables:
	float RAINTOT(time, lat, lon) ;
		RAINTOT:_FillValue = 9.96921e+36f ;
	double time(time) ;
		time:calendar = "standard" ;
		time:units = "days since 1900-01-01 00:00:00" ;
		time:_FillValue = 9.96920996838687e+36 ;
	float lat(lat, lon) ;
		lat:FieldType = 104 ;
		lat:MemoryOrder = "XY " ;
		lat:description = "LATITUDE, SOUTH IS NEGATIVE" ;
		lat:units = "degree_north" ;
		lat:stagger = "" ;
		lat:coordinates = "XLONG XLAT" ;
	float lon(lat, lon) ;
		lon:FieldType = 104 ;
		lon:MemoryOrder = "XY " ;
		lon:description = "LONGITUDE, WEST IS NEGATIVE" ;
		lon:units = "degree_east" ;
		lon:stagger = "" ;
		lon:coordinates = "XLONG XLAT" ;
}

note: I had to update all instances of netcdftime to cftime.

Units for HD17, TX and TN in Kelvins

These indices are left in units of Kelvins, instead of degrees Celsius

BUG: IcclimLogger error on windows

• icclim version: 5.0.0rc2 (?)

• Python version: ?

Description

From @DanaOdeh.

Minimal reproducible example

# !/usr/bin/env python
# coding: utf-8
import numpy
import cython
import netcdftime
import cftime
import netCDF4
import sys
import glob
import os
import time
import datetime
import dask
from netCDF4 import Dataset
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import xarray as xr
import nc_time_axis
import logging
import pytest
import setuptools
import icclim
import xclim

# In[5]

files = glob.glob('C:/.../icclimtasmax_day_CNRM-CM5_historical_r1i1p1_*.nc')

dt1 = datetime.datetime(1998,1,1)

dt2 = datetime.datetime(2005,12,31)
out_f ='SU_JJA_CNRM-CM5_historical_r1i1p1_1998-2005.nc' # OUTPUT FILE: summer season values of SU
 

icclim.index(index_name='SU', in_files=files, var_name='tasmax', time_range=[dt1, dt2], slice_mode='JJA', out_file=out_f ,logs_verbosity ='LOW')

dt1 = datetime.datetime(1998,1,1)
dt2 = datetime.datetime(2005,12,31)

out_f = 'SU_JJA_CNRM-CM5_historical_r1i1p1_1998-2005.nc' # OUTPUT FILE: summer season values of SU

icclim.index(index_name='SU', in_files=files, var_name='tasmax', time_range=[dt1, dt2], slice_mode='JJA', out_file=out_f)

Output received

Error
AttributeError                            Traceback (most recent call last)
~\AppData\Local\Temp/ipykernel_2572/579676545.py in <module>
dt2 = datetime.datetime(1986,12,31)

----> 4 icclim.index(index_name='CD', in_files='C:/Users/Dana/Desktop/icclim/pr_year_1986.nc',time_range=[dt1, dt2], var_name='pr', slice_mode='year', out_file='C:/Users/Dana/Desktop/icclim/new_pr_year_1986.nc')

C:\ProgramData\Anaconda\lib\site-packages\icclim\main.py in index(in_files, var_name, index_name, slice_mode, time_range, out_file, threshold, transfer_limit_Mbytes, callback, callback_percentage_start_value, callback_percentage_total, base_period_time_range, window_width, only_leap_years, ignore_Feb29th, interpolation, out_unit, netcdf_version, user_index, save_percentile, logs_verbosity, indice_name, user_indice)
    127     log.set_verbosity(logs_verbosity)
    128 
--> 129     log.start_message()
    130     callback(callback_percentage_start_value)
    131     if indice_name is not None:
 
C:\ProgramData\Anaconda\lib\site-packages\icclim\icclim_logger.py in start_message(self)
     62 
     63         # flake8: noqa
---> 64         time_now = time.asctime(time.gmtime()) + " " + self.timezone
     65         if self.verbosity == Verbosity.SILENT:
     66             return

AttributeError: 'IcclimLogger' object has no attribute 'timezone'

Bug in calling CWD_calculation in calc_ind.py

The call to CWD_calculation is missing the module reference.

res =CWD_calculation(**kwargs)

should be:

res =calc_indice.CWD_calculation(**kwargs)

Urgent: Only the latest documentation is available at http://icclim.readthedocs.org

But we also need the access to previous documentation for the reference netcdf attribute.

I am working on assigning Netcdf attribute values to a set of indices calculated by icclim310.
Can someone enable links to previous references?

Many thanks!

Indicator calculation on KNMI EOBS files fails.

Calculating indicator SU on the following resource fails:

http://opendap.knmi.nl/knmi/thredds/dodsC/e-obs_0.25regular/tx_0.25deg_reg_v12.0.nc

This is an OpenDAP URL (use ncdump -h on URL).

append to history attribute

The history string from ICCLIM is prepended at this point: https://github.com/tatarinova/icclim/blob/test/icclim/set_globattr.py#L75.

I think it should be appended based on: http://cf-pcmdi.llnl.gov/documents/cf-conventions/1.6/cf-conventions.html#description-of-file-contents.

Very minor change...

Icclim does not import

Hi have installed icclim and when I conda list from terminal I see it, however, when I conda it does not appear in my envrionment when I check from pycharm and doesnt import as well. I would be glad if I can be assisted with this installation as I need it for some computations. Thanks

2D Lat/Lon coordinates not supported

As reported, the icclim does not support when the input file contains spatial coordinates in 2D.
Since the icclim approach is based on processing data values over time for each grid point (station), the grid characteristics are only loaded and then rewritten to an output file. Therefore, this should be an easy fix.

Since the CMIP6 output is getting ready and the icclim is starting to be used for this purpose, we should make sure that the icclim supports fully those files.

Make RXXpTOT indices accept both "%" (ETCCDI) and "mm" (ECA&D) as output unit.

The calculation of R90pTOT and R95pTOT use the generic function RXXpTOT defined in util/calc.py. This function calculates the sum of rainfall exceeding the percentile values provided but should calculate the ratio of this relative to the total rainfall values.

I have implemented a fix which I can contribute.

Threshold input parameter for indice calculator is broken when using low transfer_limit_Mbytes

When using transfer_limit_Mbytes = 45, which puts ICCLIM in chunking mode, the custom threshold is not working.

Here is a script to reproduce:

import icclim
import datetime
files=['http://opendap.knmi.nl/knmi/thredds/dodsC/e-obs_0.25regular/tx_0.25deg_reg_v12.0.nc']

dt1=datetime.datetime(1965,1,1)
dt2=datetime.datetime(1965,12,31)
out_f = 'SU_YEAR_E-OBS_observation-opendap_r1i1p1_1965-1986.nc'

icclim.indice(indice_name='SU', in_files=files, var_name='tx', threshold=[-248.15],time_range=[dt1, dt2], transfer_limit_Mbytes=45.0, slice_mode='year', out_file=out_f)

And here is the error:
$ python testicclim.py
Loading data: chunk 1/2 ...
Loading data: chunk 2/2 ...
Traceback (most recent call last):
File "testicclim.py", line 17, in
icclim.indice(indice_name='SU', in_files=files, var_name='tx', threshold=[-248.15],time_range=[dt1, dt2], transfer_limit_Mbytes=45.0, slice_mode='year', out_file=out_f)
File "/nobackup/users/plieger/c4i_dev/wps/virtenv/lib/python2.7/site-packages/icclim-4.1.1-py2.7.egg/icclim/icclim.py", line 567, in indice
dict_threshold_indice_arr[t][:, i1_row_current_tile:i2_row_current_tile, i1_col_current_tile:i2_col_current_tile] = indice_arr_current_chunk
KeyError: -248.15

Can't use NETCDF4 (HDF5) files as input even if there is only one input file

A number of points in the code use MFDataset to open multiple input files as one dataset. However MFDataset doesn't support NETCDF4 files. It would be great to revert to just using Dataset to open files if there is only one input file. I'm aware things might change with the shift using xarray, but in the interim this is useful functionality.

[xarray] CWD; CDD; cannot be computed on xarray_new_version

For now CWD; and CDD; indices are failing with the following error when computed on xarray_new_version branch.

Error message :

"Traceback (most recent call last):
  File ""script_test.py"", line 50, in <module>
    netcdf_processing()
  File ""script_test.py"", line 39, in netcdf_processing
    icclim.indice(indice_name=indice_param,
  File ""/Users/aoun/workspace/icclim/icclim/icclim.py"", line 192, in indice
    ds = check.formatting_before_calculation(ds, var_name, indice_name, slice_mode)
  File ""/Users/aoun/workspace/icclim/icclim/util/check.py"", line 40, in formatting_before_calculation
    return subset.vectorize(ds, var_name, indice_name, slice_mode)    
  File ""/Users/aoun/workspace/icclim/icclim/subset.py"", line 72, in vectorize
    dat[np.arange(len(monthA[month])),:] = ds_year[indice_name].values[monthA[month],:]
ValueError: shape mismatch: value array of shape (31,8,64,128) could not be broadcast to indexing result of shape (31,64,128)"

Simple test suite for ICCLIM

Implement a simple test framework/script for simple system/regression tests of ICCLIM. Initial test cases would be,

• Run a set of indices to produce output files without crashing
• Compare produced output with reference data set, either from earlier tests or from independently generated indices.

My initial idea is a simple test script with test cases defined in a separate file and to wrap it all up using the Robot Framwork

ZeroDivisionError

Hi! I run version 4.2.11 (although, by the way, I downloaded the latest package, named 4.2.12). It works fine with some netcdf files, until I get this fatal error with files from a specific model.
Traceback (most recent call last): File "ICCLIMscriptPI.py", line 36, in <module> icclim.indice(in_files=ncfile[0], var_name=var_name, out_file=out_file, indice_name=indice_name, slice_mode='DJF', callback = callback.defaultCallback) File "/home/scussoli/.local/lib/python2.7/site-packages/icclim-4.2.11-py2.7.egg/icclim/icclim.py", line 678, in indice user_indice=user_indice) ## tuple: (dt_centroid_arr, dt_bounds_arr, indice_arr) File "/home/scussoli/.local/lib/python2.7/site-packages/icclim-4.2.11-py2.7.egg/icclim/icclim.py", line 870, in get_indice_from_dict_temporal_slices percentage_slice = (callback_percentage_total*1.0)/(nb_t_slices*nb_chunks) ZeroDivisionError: float division by zero
A snippet of the input nc file and the python script can be found at: https://surfdrive.surf.nl/files/index.php/s/wu7MFtLwoVGv560
Thanks!

Approach for time_units attribute

This is more a question than potential issue.
In the icclim, the time@units is read only for the first input file, if multiple input files are loaded, and assumed to be the same for all files.

How much this might affect the accuracy and performance computing of percentiles?

Many thanks,

functions of the calc_indice.py module: passed array could be masked or filled

who: @bekozi

To allow masked arrays to be used in addition to filled arrays.

Technical implementation of adding metadata

A sub issue is to add reference data in the output file.

Data type of SHORT: not supported?

Another reported issue. If this is true, then the option for all data types, excepts string of course, should be introduced.
I've used icclim on float and double but never on short data so I do not recall myself this issue.

Collaboration opportunity?

Hello Christian et al,

I recently became aware of this project as a result of a search for climate indices in Python. I have also written a Python package for climate indices containing indices mostly applicable to drought/climate monitoring. I'm interested in collaborating if possible in case my code can be useful to your project. It seems that you're using the same underlying libraries, i.e. numpy, NetCDF, etc. so it might not be too heavy of a lift.

Coincidentally it happens that I will be visiting a friend in Toulouse in late May and the first half of June, so I will be nearby in case that may facilitate things. If this is of interest please contact me. Thanks!

--James

Python 3 compatibility?

ocgis is supporting Python 3.5+ in the next release. I naively tested against an icclim 4.2.5 version built with Python 3.6 and encountered issues. Does icclim have plans to support Python 3? Python 2.7 will still be supported in ocgis. I made an attempt to fix some of the basic syntax issues but received some errors deep in the calculation code which I couldn't parse easily.

Minor output netCDF file and meta data issues

• Index variable name in output NetCDF should be all set to small letters (as previously agreed). For example, TN10p -> tn10p in output file.
• Index variable attribute 'units' in output NetCDF is associated to 'days' even when input argument is chosen to be '%'
• Index values for which a given condition (by its definition) is not satisfied are set to missing value. But this should be corrected in order to make distinction between grid points with no input data and those where it is not possible for an index to be defined.
• icclim.py, line: 394: var_units == 'Celcius' to be replaced by var_units == 'Celsius'

libraryC import

For CSU_calculate and CFD_calculate, I got the following exception:

File "/usr/local/lib/python2.7/dist-packages/icclim-0.0.1-py2.7.egg/icclim/calc_indice.py", line 279, in CSU_calculation
C_find_max_len_consec_sequence_3d = libraryC.find_max_len_consec_sequence_3d
NameError: global name 'libraryC' is not defined

AttributeError in the indice function

I am relatively new to python so may be overlooking something quite obvious (apologies in that case), but no matter what input nc file I use, when calling the icclim.indice function, it 'begins execution' and the immediately raises an
AttributeError: NetCDF: Attribute not found
It gets stuck in the icclim.py file at
line 245, in indice fill_val = ncVar._FillValue.astype('float32') # fill_value must be the same type as "ind_type", i.e. 'float32'
File "netCDF4/_netCDF4.pyx", line 3913, in netCDF4._netCDF4.Variable.__getattr__ File "netCDF4/_netCDF4.pyx", line 3724, in netCDF4._netCDF4.Variable.getncattr File "netCDF4/_netCDF4.pyx", line 1188, in netCDF4._netCDF4._get_att File "netCDF4/_netCDF4.pyx", line 1638, in netCDF4._netCDF4._ensure_nc_success
I call the function this way:
icclim.indice(in_files=ncfile_LIG, var_name='PRECT', out_file=out_file, indice_name='RX5day', slice_mode='year', callback = callback.defaultCallback)
The input nc files I tried are fine for all other purposes. I guess there's something wrong with the fill value of (all) my variables (type: float32), but can't seem to find a solution.
Could you assist please?

indice function does not work for some slice_mode options

Hi!
I have been working with the package for a while.
Now, with some files, it would calculate the RX5day or RX1day indices for the 'year' or 'DJF' slice modes, but not for the rest of the seasons: MAM, JJA, SON. It throws this error:
Traceback (most recent call last): File "<stdin>", line 1, in <module> File "icclim/icclim.py", line 645, in indice time_range=time_range) File "icclim/time_subset.py", line 300, in get_dict_temporal_slices dtt_num_i = util_dt.date2num(dt_arr_subset_i[-1], calend, tunits) + seconds_per_day IndexError: index -1 is out of bounds for axis 0 with size 0
I tried multiple times. Note that with some files it does work, but with others it won't, and I can't pinpoint to the difference between the two files, as they come from the same climate model simulation, and are of similar size.
Please could you assist? Thank you so much.

ENH: Run doc examples in the CI

• icclim version: 5.0.0rc2

• Python version: all

Description

We have quite a few example in the documentation but it is quite hard to test them by hand with each new version of icclim.

It seems feasible to run all examples, both from docstrings and from the actual doc when we open a new pull request, I believe it is the duty of doctest.

This would make the maintenance of these example much easier and would ensure they are not broken as reported for example in #65