GithubHelp home page GithubHelp logo

juliaclimate / climatetools.jl Goto Github PK

View Code? Open in Web Editor NEW
116.0 5.0 18.0 8.76 MB

Climate science package for Julia

Home Page: https://juliaclimate.github.io/ClimateTools.jl/stable/

License: Other

Julia 100.00%
climate-indices climate-science netcdf-files julia parallel-computing multiple-threads timeseries climate-analysis bias-correction

climatetools.jl's Introduction

Climate analysis tools in Julia

Project Status: Active – The project has reached a stable, usable state and is being actively developed. Build status Build status codecov

DOI chat MIT license

Latest release: GitHub release (latest SemVer)

Documentation

Overview

Note. Compatible with Julia 1.9 and higher

ClimateTools.jl is a collection of commonly-used tools in Climate science. Basics of climate field analysis are covered, with some forays into exploratory techniques associated with climate scenarios design. The package is aimed to ease the typical steps of analysis climate models outputs and gridded datasets (support for weather stations is a work-in-progress).

ClimateTools.jl is registered on METADATA.jl and can be added and updated with Pkg commands. See installation documentation for detailed installation instructions and Python's dependencies (for mapping features).

Climate indices and bias correction functions are coded to leverage the use of multiple threads. To gain maximum performance, use (bash shell Linux/MacOSX) export JULIA_NUM_THREADS=n, where n is the number of threads. To get an idea of the number of threads you can use type (in Julia) Sys.THREADS. This is especially useful for bias correction.

Contributors

If you'd like to have other climate indices coded, please, submit them through a Pull Request! I'd be more than happy to include them. Alternatively, provide the equation in Issues.

Features

  • Extraction. manipulation and visualization of CF-compliant netCDF datasets
  • Climate indices from The joint CCl/CLIVAR/JCOMM Expert Team (ET) on Climate Change Detection and Indices (ETCCDI) as well as custom climate indices. See list.
  • Regridding of a datasets onto another grid
  • Post-processing of climate timeseries using Quantile-Quantile mapping method (cf. Themeßl et al. 2012, Piani et al. 2010)
  • Post-processing of for extreme values (Roy et al. 2023)

Getting started

Note. More in-depth documentation is provided in the official documentation (Links: stable/latest).

using ClimateTools

Reading a NetCDF file

The entry point of ClimateTools is to load data with the load function. Optional polygon clipping feature is available. By providing such polygon, the load function returns a ClimGrid with grid points contained in the polygon.

C = load(filename::String, vari::String; poly::Array, data_units::String, start_date::Tuple, end_date::Tuple)

load returns a ClimGrid type. Using the optional poly argument, the user can provide a polygon and the returned ClimGrid will only contains the grid points inside the provided polygon. For some variable, the optional keyword argument data_units can be provided. For example, precipitation in climate models are usually provided as kg/m^2/s. By specifying data_units = mm, the load function returns accumulation at the data time resolution. Similarly, the user can provide Celsius as data_units and load will return Celsius instead of Kelvin.

The ClimGrid is a in-memory representation of a CF-compliant netCDF file for a single variable.

struct ClimGrid
  data::AxisArray # labeled axis
  longrid::AbstractArray{N,2} where N # the longitude grid
  latgrid::AbstractArray{N,2} where N # the latitude grid
  msk::Array{N, 2} where N
  grid_mapping::Dict # bindings of native grid
  dimension_dict::Dict
  model::String
  frequency::String
  experiment::String
  run::String
  project::String # CORDEX, CMIP5, etc.
  institute::String
  filename::String
  dataunits::String
  latunits::String # of the coordinate variable
  lonunits::String # of the coordinate variable
  variable::String # Type of variable (i.e. can be the same as "var", but it is changed when calculating indices)
  typeofvar::String # Variable type (e.g. tasmax, tasmin, pr)
  typeofcal::String # Calendar type
  timeattrib::Dict # Time attributes
  varattribs::Dict # Variable attributes
  globalattribs::Dict # Global attributes

end

Subsetting

Further subsets can be done in the temporal and spatial domains. spatialsubset function acts on ClimGrid type and subset the data using a user polygon. The function returns another ClimGrid.

C = spatialsubset(C::ClimGrid, poly:Array{N, 2} where N)

Temporal subset of the data is done with temporalsubset function, which returns a continuous timeserie between startdate and enddate.

C = temporalsubset(C::ClimGrid, startdate::Tuple, enddate::Tuple)

Resampling is available with the resample, which returns a given period for each year (e.g. only summer months).

C = resample(C::ClimGrid, startmonth::Int, endmonth::Ind)
C = resample(C::ClimGrid, season::String) # hardcoded seasons -> "DJF", "MAM", "JJA" and "SON"

Mapping the ClimGrid type

Mapping climate information can be done by using mapclimgrid.

mapclimgrid(C::ClimGrid; region = "World")

Which should return the time average of ClimGrid C over the world region.

Precipitation example

Note that if the ClimGrid data structure has 3 dimensions (time x longitude x latitude) the mapclimgrid function makes a time-average (i.e. climatological mean). Right now, there are a growing list of hardcoded regions (see help section of mapclimgrid function) and the default auto which use the maximum and minimum of the lat-long coordinates inside the ClimGrid structure. The user can also provide a polygon(s) and the mapclimgrid function will clip the grid points outside the specified polygon. Another option is to provide a mask (with dimensions identical to the spatial dimension of the ClimGrid data) which contains NaN and 1.0 and the data inside the ClimGrid struct will be clipped with the mask. Other regions will be added in the future, as well as the option to send a custom region defined by a lat-lon box.

Indices

More than 20 climate indices are available in the package, such as the annual number of tropical nights, annual maximum and minimum, etc. You can calculate such indices simply with:

ind = annualmax(C::ClimGrid)

Which returns another ClimGrid. You can also map this ClimGrid with the mapclimgrid function and returns the climatological mean of the annual maximum (e.g. daily precipitation in the example below). From the figure, we clearly sees the monsoon regions (India) and region with wind-driven precipitations (e.g. western sides of the oceans).

A list of indices can be found in the documentation and in the functions.jl source code.

Precipitation example

Climate indices can easily be developed by following the source code or looking at the available metadata inside a ClimGrid.

Interpolation

A typical step in climate analysis is to interpolate a given grid onto another grid. ClimateTools provides such a tool by wrapping Scipy griddata function. It is intended for visualization or as a 1st step before bias-correcting the ClimGrid dataset.

The following command will interpolate the data contained in ClimGrid A into the coordinates of ClimGrid B and returns a new ClimGrid C which contains the interpolated data of A into the grid of B.

C = regrid(A::ClimGrid, B::ClimGrid)

It is also possible to interpolate a ClimGrid onto specified longitude and latitude vectors.

C = regrid(A::ClimGrid, lon::AbstractArray{N, 1}, lat::AbstractArray{N, 1})

Bias-correction

See Documentation.

Merging ClimGrids

Sometimes, the timeseries are split among multiple files (e.g. climate models outputs). To obtain the complete timeseries, you can merge 2 ClimGrid. The method is based on the merging of two AxisArrays and is overloaded for the ClimGrid type.

C = merge(C1::ClimGrid, C2::ClimGrid)

Exporting

It is possible to export to a netCDF file with the command write

write(C::ClimGrid, filename::String)

TO-DO

  • Dashboard tool. This will return the main characteristics of a ClimGrid: maps of minimum, maximum and mean climatological values, seasonal cycle, timeseries of annual maximum, minimum and mean values, etc...
  • Create a WeatherStation type.
  • Add a more complex quantile-quantile mapping technique, combining extreme value theory and quantile-quantile standard technique

climatetools.jl's People

Contributors

balinus avatar elnol avatar femtocleaner[bot] avatar fidelthomet avatar gaelforget avatar github-actions[bot] avatar juliatagbot avatar mlubin avatar pitmonticone avatar staticfloat avatar tkelman avatar zeitsperre avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar

climatetools.jl's Issues

Load NC file fails though works with ncread()

I'm getting a "NCDatasets.NetCDFError(-49, "NetCDF: Variable not found")" error when attempting to use ClimateTools.load() to load a .nc file. The same file loads fine with the ncread() function from the NetCDF package. I'm using julia v1.0. Thoughts?

difficulty installing climateplots

I first installed ClimateTools via Pkg.add (got v0.14.5 somehow) then proceeded to install ClimatePlots as explained in the docs. But this approach lead to the error message below.

(v1.3) pkg> add ClimatePlots
 Resolving package versions...
ERROR: Unsatisfiable requirements detected for package ClimatePlots [b71d359b]:
 ClimatePlots [b71d359b] log:
 ├─possible versions are: [0.1.0, 0.2.0-0.2.2] or uninstalled
 ├─restricted to versions * by an explicit requirement, leaving only versions [0.1.0, 0.2.0-0.2.2]
 └─restricted by compatibility requirements with ClimateTools [4f4ee721] to versions: uninstalled — no versions left
   └─ClimateTools [4f4ee721] log:
     ├─possible versions are: [0.9.0, 0.10.0-0.10.1, 0.11.0-0.11.3, 0.12.0, 0.13.0, 0.14.0-0.14.5, 0.15.0, 0.16.0, 0.17.0, 0.18.0-0.18.1] or uninstalled
     └─restricted to versions 0.14.5 by an explicit requirement, leaving only versions 0.14.5

update:

I went back and, instead of using Pkg.add, I cloned the repo and used Pkg.dev. This resolved the above error and now I have both ClimateTools and ClimatePlots compiled. Maybe all that's needed is an update to the docs?

Installation and Basemap dependency

I tried to install ClimateTools.jl which worked with ]add ClimateTools.
However, at my first
using ClimateTools
it seems like it tries to install basemap with Conda, which fails because apparently basemap is not compatible with python3.9.

[ Info: Installing mpl_toolkits.basemap via the Conda basemap package...
[ Info: Running `conda install -y basemap` in root environment
Collecting package metadata (current_repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Solving environment: failed with repodata from current_repodata.json, will retry with next repodata source.
Collecting package metadata (repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Solving environment: / 
Found conflicts! Looking for incompatible packages.
This can take several minutes.  Press CTRL-C to abort.
failed                                                                                                                              

UnsatisfiableError: The following specifications were found
to be incompatible with the existing python installation in your environment:

Specifications:

  - basemap -> python[version='>=2.7,<2.8.0a0|>=3.6,<3.7.0a0|>=3.7,<3.8.0a0|>=3.8,<3.9.0a0']

Your python: python=3.9

If python is on the left-most side of the chain, that's the version you've asked for.
When python appears to the right, that indicates that the thing on the left is somehow
not available for the python version you are constrained to. Note that conda will not
change your python version to a different minor version unless you explicitly specify
that.

From what I read it sounded like basemap was not required by ClimateTools.jl so I don't understand why this happens.

merge() should have option to ensure temporal consistency

As we discovered earlier, a feature/bug that can occur when looping through NetCDFs is that datetime variables are not respected. ie:

merge(file2017, file2018): Dates following 2017 (Jan-Dec) -> 2018 (Jan-Dec)
and
merge(file2018, file2017): dates following 2018 (Jan-Dec) -> 2017 (Jan-Dec)

I can't think of a reason for wanting the second option, but maybe an internal check ("time_consistency"?) that defaults to boolean true and launches a method ensuring that data sets follow each other would be a good addition. I can see it being useful for merging badly named datasets or for instances of merging data from different projects (again, not sure why one would do that but anyway).

pycall api update

PyCall 1.90.0 is now released, which change o[:foo] and o["foo"] to o.foo and o."foo", respectively, for python objects o; see also JuliaPy/PyCall.jl#629.

The old getindex methods still work but are deprecated, so you'll want to put out a new release that uses the new methods and REQUIREs PyCall 1.90.0 to avoid having zillions of deprecation messages.

UndefVarError: mapclimgrid not defined

Hello! I'm trying to reproduce your example here but when I try to run

mapclimgrid(model, region = "Quebec")

I get the following error:

UndefVarError: mapclimgrid not defined

Any idea what I'm doing wrong?

Dashboard tool

Add a dashboard tool that summarize a given ClimGrid.

  • annual cycle
  • map
  • timeserie
  • annual min
  • annual max

ClimateTools adjusts the dependecies versions of installed packages, causing other packages to precompile incorrectly.

ClimateTools adjusts the dependecies versions of installed packages, causing other packages to precompile incorrectly. For example, ClimateTools would drop the version of the StatsBase package from 0.33.21 to 0.32.2, while pairwise! was only added at 0.33.8. This causes the problem of not finding the pairwise! function. It also caused errors in the pre-compilation of the Plots package.
What is the solution?

`julia> using StatsBase
help?> pairwise!
search:

Couldn't find pairwise!
Perhaps you meant pairs
No documentation found.

Binding pairwise! does not exist.`

julia> using Plots [ Info: Precompiling Plots [91a5bcdd-55d7-5caf-9e0b-520d859cae80] ERROR: LoadError: MethodError: no method matching floatrange(::Type{Float64}, ::Int64, ::Int64, ::Float64, ::Int64) Closest candidates are: floatrange(::Type{T}, ::Integer, ::Integer, ::Integer, ::Integer) where T at C:\Users\Administrator\AppData\Local\Programs\Julia-1.7.1\share\julia\base\twiceprecision.jl:370 floatrange(::AbstractFloat, ::AbstractFloat, ::Real, ::AbstractFloat) at C:\Users\Administrator\AppData\Local\Programs\Julia-1.7.1\share\julia\base\twiceprecision.jl:384 Stacktrace: [1] floatrange(a::Float64, st::Float64, len::Float64, divisor::Float64) @ Base .\twiceprecision.jl:392 [2] histrange(lo::Float64, hi::Float64, n::Int64, closed::Symbol) @ StatsBase C:\Users\Administrator\.julia\packages\StatsBase\548SN\src\hist.jl:99 [3] histrange(v::Vector{Float64}, n::Int64, closed::Symbol) @ StatsBase C:\Users\Administrator\.julia\packages\StatsBase\548SN\src\hist.jl:39 [4] _hist_edge @ C:\Users\Administrator\.julia\packages\Plots\CUQan\src\recipes.jl:798 [inlined] [5] #212 @ C:\Users\Administrator\.julia\packages\Plots\CUQan\src\recipes.jl:811 [inlined] [6] map @ .\tuple.jl:222 [inlined] [7] _hist_edges @ C:\Users\Administrator\.julia\packages\Plots\CUQan\src\recipes.jl:811 [inlined] [8] _make_hist(vs::Tuple{Vector{Float64}, Vector{Float64}}, binning::Int64; normed::Bool, weights::Nothing) @ Plots C:\Users\Administrator\.julia\packages\Plots\CUQan\src\recipes.jl:830 [9] macro expansion @ C:\Users\Administrator\.julia\packages\Plots\CUQan\src\recipes.jl:952 [inlined] [10] apply_recipe(plotattributes::AbstractDict{Symbol, Any}, #unused#::Type{Val{:histogram2d}}, x::Any, y::Any, z::Any) @ Plots C:\Users\Administrator\.julia\packages\RecipesBase\z10lo\src\RecipesBase.jl:300 [11] _process_seriesrecipe(plt::Any, plotattributes::Any) @ RecipesPipeline C:\Users\Administrator\.julia\packages\RecipesPipeline\XxUHt\src\series_recipe.jl:50 [12] _process_seriesrecipes!(plt::Any, kw_list::Any) @ RecipesPipeline C:\Users\Administrator\.julia\packages\RecipesPipeline\XxUHt\src\series_recipe.jl:27 [13] recipe_pipeline!(plt::Any, plotattributes::Any, args::Any) @ RecipesPipeline C:\Users\Administrator\.julia\packages\RecipesPipeline\XxUHt\src\RecipesPipeline.jl:99 [14] _plot!(plt::Plots.Plot, plotattributes::Any, args::Any) @ Plots C:\Users\Administrator\.julia\packages\Plots\CUQan\src\plot.jl:232 [15] #plot#152 @ C:\Users\Administrator\.julia\packages\Plots\CUQan\src\plot.jl:107 [inlined] [16] #histogram2d#477 @ C:\Users\Administrator\.julia\packages\RecipesBase\z10lo\src\RecipesBase.jl:427 [inlined] [17] var"##10#282"() @ Plots C:\Users\Administrator\.julia\packages\Plots\CUQan\src\precompilation.jl:15 [18] top-level scope @ C:\Users\Administrator\.julia\packages\Plots\CUQan\src\precompilation.jl:25 [19] eval @ .\boot.jl:373 [inlined] [20] eval @ C:\Users\Administrator\.julia\packages\Plots\CUQan\src\Plots.jl:1 [inlined] [21] _broadcast_getindex_evalf @ .\broadcast.jl:670 [inlined] [22] _broadcast_getindex @ .\broadcast.jl:643 [inlined] [23] getindex @ .\broadcast.jl:597 [inlined] [24] copyto_nonleaf!(dest::Vector{Nothing}, bc::Base.Broadcast.Broadcasted{Base.Broadcast.DefaultArrayStyle{1}, Tuple{Base.OneTo{Int64}}, typeof(Plots.eval), Tuple{Base.Broadcast.Extruded{Vector{Expr}, Tuple{Bool}, Tuple{Int64}}}}, iter::Base.OneTo{Int64}, state::Int64, count::Int64) @ Base.Broadcast .\broadcast.jl:1055 [25] copy(bc::Base.Broadcast.Broadcasted{Base.Broadcast.DefaultArrayStyle{1}, Tuple{Base.OneTo{Int64}}, typeof(Plots.eval), Tuple{Vector{Expr}}}) @ Base.Broadcast .\broadcast.jl:907 [26] materialize @ .\broadcast.jl:860 [inlined] [27] macro expansion @ C:\Users\Administrator\.julia\packages\Plots\CUQan\src\precompilation.jl:32 [inlined] [28] macro expansion @ C:\Users\Administrator\.julia\packages\SnoopPrecompile\1XXT1\src\SnoopPrecompile.jl:69 [inlined] [29] (::Plots.var"#589#590")() @ Plots C:\Users\Administrator\.julia\packages\Plots\CUQan\src\precompilation.jl:29 [30] withenv(f::Plots.var"#589#590", keyvals::Pair{String, String}) @ Base .\env.jl:172 [31] macro expansion @ C:\Users\Administrator\.julia\packages\Plots\CUQan\src\precompilation.jl:28 [inlined] [32] macro expansion @ C:\Users\Administrator\.julia\packages\SnoopPrecompile\1XXT1\src\SnoopPrecompile.jl:119 [inlined] [33] top-level scope @ C:\Users\Administrator\.julia\packages\Plots\CUQan\src\precompilation.jl:4 [34] include(mod::Module, _path::String) @ Base .\Base.jl:418 [35] include(x::String) @ Plots C:\Users\Administrator\.julia\packages\Plots\CUQan\src\Plots.jl:1 [36] top-level scope @ C:\Users\Administrator\.julia\packages\Plots\CUQan\src\Plots.jl:204 [37] include @ .\Base.jl:418 [inlined] [38] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt64}}, source::Nothing) @ Base .\loading.jl:1318 [39] top-level scope @ none:1 [40] eval @ .\boot.jl:373 [inlined] [41] eval(x::Expr) @ Base.MainInclude .\client.jl:453 [42] top-level scope @ none:1 in expression starting at C:\Users\Administrator\.julia\packages\Plots\CUQan\src\precompilation.jl:3 in expression starting at C:\Users\Administrator\.julia\packages\Plots\CUQan\src\Plots.jl:1 ERROR: Failed to precompile Plots [91a5bcdd-55d7-5caf-9e0b-520d859cae80] to C:\Users\Administrator\.julia\compiled\v1.7\Plots\jl_AB42.tmp. Stacktrace: [1] error(s::String) @ Base .\error.jl:33 [2] compilecache(pkg::Base.PkgId, path::String, internal_stderr::IO, internal_stdout::IO, ignore_loaded_modules::Bool) @ Base .\loading.jl:1466 [3] compilecache(pkg::Base.PkgId, path::String) @ Base .\loading.jl:1410 [4] _require(pkg::Base.PkgId) @ Base .\loading.jl:1120 [5] require(uuidkey::Base.PkgId) @ Base .\loading.jl:1013 [6] require(into::Module, mod::Symbol) @ Base .\loading.jl:997
My julia environment:
`
julia> versioninfo
versioninfo (generic function with 2 methods)

julia> versioninfo()
Julia Version 1.7.1
Commit ac5cc99908 (2021-12-22 19:35 UTC)
Platform Info:
OS: Windows (x86_64-w64-mingw32)
CPU: Intel(R) Core(TM) i7-8650U CPU @ 1.90GHz
WORD_SIZE: 64
LIBM: libopenlibm
LLVM: libLLVM-12.0.1 (ORCJIT, skylake)
Environment:
JULIA_PKG_SERVER = https://mirrors.bfsu.edu.cn/julia/static
`

TagBot trigger issue

This issue is used to trigger TagBot; feel free to unsubscribe.

If you haven't already, you should update your TagBot.yml to include issue comment triggers.
Please see this post on Discourse for instructions and more details.

If you'd like for me to do this for you, comment TagBot fix on this issue.
I'll open a PR within a few hours, please be patient!

difficulty reading CF compliant files

After loading one of my files via Panoply to verify that there was nothing wrong with it (see below) I tried the model = load(gcm_files, "tasmax", poly=poly_reg) example and got ERROR: Manually verify x/lat dimension name.

Taking a look in the code I see that getdim_lat relies on a list of hard coded names. I thought that the more general approach was to rely on long_name + units. Not sure what to suggest -- adding to the hard coding list would be a short term fix just for me...

  lon_c   (720)
    Datatype:    Float64
    Dimensions:  lon_c
    Attributes:
     units                = degrees_east
     long_name            = longitude

Screen Shot 2020-02-28 at 4 03 53 PM

Problem with mapclimgrid()

CONTEXT :
I'm running on Fedora 29, I use anaconda3 as my python distribution and Julia 1.1 binaries inside of the Atom IDE. My .nc test files are from CMIP5, tasmax variable. I also reproduced this error on Windows 10.

PROBLEM :
Whenever I try to call the mapclimgrid() function, I get the following stacktrace :

ERROR: LoadError: ArgumentError: ref of NULL PyObject
Stacktrace:
[1] getindex(::PyCall.PyObject, ::String) at /home/feanor/.julia/packages/PyCall/0jMpb/src/PyCall.jl:284
[2] getindex at /home/feanor/.julia/packages/PyCall/0jMpb/src/PyCall.jl:294 [inlined]
[3] #mapclimgrid#37(::String, ::Bool, ::Array{Any,1}, ::Array{Any,1}, ::Array{Any,1}, ::Array{Any,1}, ::Function) at /home/feanor/.julia/packages/ClimateTools/W7g0z/src/mapping.jl:184
[4] mapclimgrid() at /home/feanor/.julia/packages/ClimateTools/W7g0z/src/mapping.jl:130
[5] top-level scope at none:0

I'm new to Julia and I can't solve or isolate the error.
Thanks for the help and the super cool package :)

Permutation in load function is too costly

There is a permutation of the data array in load function. Originally, this was done to optimize climate indices calculations which were faster when the time dimension was the 1st dimension. However, this prevents loading bigger datasets such as Daymet.

Solution is to simply get rid of the permutation and changes all AxisArrays construction and indices loops.

Extraction should fallback on NCDatasets.jl types

Context
When ClimateTools was designed (3 years ago already!) there was not a lot of options for Geographic types and quite less for climate data. From this perspective, the ClimGrid type was born. However, there is a lack of performance for the extraction of the data from netCDF files but also no support for on-disk datasets (data too large for the available RAM).

Challenge
The challenge will be to align the ClimGrid type with the CFVariable available in NCDatasets.jl. So, either we simply scratch the ClimGrid type found in ClimateBase and replace it with CFVariable (or NCDataset?). There is now some work done in NCDatasets.jl for multiple files dataset and datasets that are too large to fit in RAM and we perhaps should now look into it for a 1.0 release of ClimateTools.

Why
Now that the extraction ecosystem is mature, ClimateTools should now concentrate on climate analysis algorithms (bias correction, etc) and not on the extraction of datasets.

Idea
The API between ClimateTools and NCDatasets.jl should reside in ClimateBase. I'm also posting over there.

Any help, idea would be appreciated!

Seel also: JuliaClimate/ClimateBase.jl#2

Switching to Cartopy - Deprecate Basemap

Basemap will soon be no longer supported. And it's also a mess with Python 3.6+. Once ClimateTools is splitted (ClimateTools/ClimateMaps), I strongly suggest we port the code to use Cartopy before adding any new features in ClimateMaps.

Better yet would be to develop a strong support for PlotlyJS.jl. Maps are awesome, but this would involve unloading the projection part to GDAL.jl and then plot on PlotlyJS.jl.

Support for climatologies

Tried loading a few climatologies from netCDF and run into an issue with date incrementation. The example below is a climatology of 12 months for the normal period of 2041 to 2070. From ncdump:

dimensions:
time = UNLIMITED ; // (12 currently) lat = 3085 ; lon = 7747 ;
variables:
int time(time) ; time:axis = "T" ; time:units = "months since 2055-1-1" ; time:long_name = "Time" ; time:standard_name = "time" ; time:calendar = "standard" ; time:climatology = "climatology_bounds" ; time:climo_period = "2041/01/01 - 2070/12/31" ;
double climatology_bounds ;
float lat(lat) ; lat:axis = "Y" ; lat:units = "degrees_north" ; lat:long_name = "latitude" ; lat:standard_name = "latitude" ;
float lon(lon) ; lon:axis = "X" ; lon:units = "degrees_east" ; lon:long_name = "longitude" ; lon:standard_name = "longitude" ;
float tasmax(time, lat, lon) ; tasmax:_FillValue = -9999.f ; tasmax:units = "deg C" ; tasmax:long_name = "Maximum Air Temperature" ; tasmax:standard_name = "max_air_temperature" ; tasmax:cell_methods = "time: mean over years" ; tasmax:comments = "Monthly average as calculated across the climatology." ;

:time = "2055-1-15 2055-2-15 2055-3-15 2055-4-152055-5-152055-6-152055-7-152055-8-15 2055-9-15 2055-10-15 2055-11-152055-12-15" ;
:climatology_bounds = "2055-1-1 2055-1-31 2055-2-1 2055-2-28 2055-3-1 2055-3-31 2055-4-1 2055-4-30 2055-5-1 2055-5-31 2055-6-1 2055-6-30 2055-7-1 2055-7-31 2055-8-1 2055-8-31 2055-9-1 2055-9-31 2055-10-1 2055-10-31 2055-11-1 2055-11-30 2055-12-1 2055-12-31"

On load, it seems to be read as days starting from Jan 1, 2055.

ClimGrid struct with data:
3-dimensional AxisArray{Float32,3,...} with axes:
:time, Date[2055-01-01, 2055-01-02, 2055-01-03, 2055-01-04, 2055-01-05, 2055-01-06, 2055-01-07, 2055-01-08, 2055-01-09, 2055-01-10, 2055-01-11, 2055-01-12]

Is this a matter of needing to include support for when time:units is "months"?

Replace nc2julia with NCDataset.jl implementation

nc2julia is rather a crude implementation of converting a netCDF file to a Julia struct. Since there are numerous type of files to take care, its probably better to use work done in NCDataset.jl package and convert the output of their Dataset type into the ClimGrid type of ClimateTools.jl and keep the polygon/shapefile features of the current nc2julia function.

Add new regions

Would be useful to have the following region for mapping.

  • Africa
  • Middle-East
  • Russia
  • China
  • Oceania
  • Australia
  • Pacific
  • Indian ocean
  • Atlantic ocean

Remove Unitful integration

After some use-case, I find that having Unitful around is way more work than anticipated. I think that dropping this feature will lead to an easier package to maintain.

Add interpolation for ClimGrid type

For the bias correction method, it if often useful to have a common grid between the reference and the simulated time series. As such, an interpolation function for the ClimGrid type is needed.

Export options

  • Export ClimGrid to netCDF file
  • Export ClimGrid to GeoTIFF/ASCII grid
  • Export Maps to PNG
  • Export data of a ClimGrid to CSV

Keep only climate indices that takes a ClimGrid as an argument

I think they are not needed in the end. People using the package will not extract the raw data from the ClimGrid and then call the climate indices functions and alternatively, people that have a barebone data array won't load ClimateTools just for the indices.

Advantages : much easier to maintain!

[DOC] Improve regrid documentation

Hey there,

I would like to use the regridding functionality to convert my LatLon grid data (1 degree spacing in both lon and lat) to an equal area grid. I am now writing code that produces an approximate equal area grid (which I will then immediately PR here as this seems super duper useful for me :D ).

But I find the regird function poorly documented. Firstly, what are the options for method? Secondly, may I suggest to have the regrid docstring into the "interpolation" page instead of "index"? It makes much more sense. Thirdly, this:

C = regrid(A::ClimGrid, lon::AbstractArray{N, T} where N where T, lat::AbstractArray{N, T} where N where T; dimx=[], dimy=[], method::String="linear", min=[], max=[])

is way too verbose as a docstring, and I can't imagine the information t::AbstractArray{N, T} where N where T is actually useful or relevant for the function. Fourthly, I simply don't understand the call signature regrid(A::ClimGrid, lon, lat). Which field is regrided here? The field A into the given LonLat grid?

Lastly, it should also be discussed whether grids are expected in degrees or kilometers (as some people also use kilometers).

Include indices from xclim

A feature I would like to see built into ClimateTools would be a few methods for performing agroclimatic indices. Some ideas:

  • Growing Degree Days (standard degrees above base in Celsius)
  • Biologically Effective Degree Days (Integrating daily min and max with baselines and upper limits)

We could also consider having preset indicators for Heating/Cooling Degree Days (degrees below 18 and above 18, respectively) or GDDs above 0, 5, 10 as they can have engineering and agroclimate purposes, respectively.

I'll be creating a new branch to work on this.

Merging with ClimateEasy.jl

Hey @Balinus, just wanted to let you know that I've put up one of my repositories ClimateEasy.jl into JuliaClimate. However, I think eventually to prevent confusion I should merge the repository into ClimateTools.jl, but my way of coding might be quite different from yours.

Could you let me have a look and see what you think about merging the two repositories and how we should go about doing it?

Spatial averaging (in plot function)

Hi all,

I am not sure if this is the right repository since I just found the idea of outsourcing the topic of mapping/plotting to an own package (ClimateMapping).

Nevertheless, if used for Gaussian lon x lat grids the plot function needs to account for converging grid point at higher latitudes when doing a spatial average. When used to plot a global mean, the current function overemphasises polar regions. I already have a latitudinal weighted version, but wanted to open the issue before making a pull request. Furthermore, I hesitate since my version is more or less only valid on Gaussian grids.

Any ideas, any thoughts on the issue of spatial averaging in the plot function. The area-weighted averaging could also be implemented as a separate function, but then using the plot function for a global field still gives an invalid figure.

Best,

  • ikroener

Colormaps - more than 8 colors!

Find a way to show more than 8 colors in maps! Still haven't found a way to go higher than the base colormaps. There should be an easy way to interpolate between these colors. The plotting is done using PyPlot/basemap, so I guess the solution should be explored in Python api.

Dependency updates needed

  1. Currently ClimateTools.jl conflicts with ClimateBase.jl > 0.6 cause 0.7 removed the ClimGrid type and functionalities.

In JuliaClimate/Notebooks, I wanted to demo ClimateTools.jl and ClimateBase.jl together in this notebook as the ClimateBase.jl docs state that they are related.

But it seems that I cannot simply do that, owing in part to the conflict noted above.

For now, a simple solution to this would be to copy the ClimGrid type and functionalities that were removed from ClimateBase.jl in 0.7 back into ClimateTools.jl, and then remove the ClimateBase.jl 0.6 dependency in ClimateTools.jl

See https://github.com/gaelforget/ClimateTools.jl/tree/update_dependencies_gf1

This prompted me to look at ClimateTools dependencies in more detail.

  1. InverseDistanceWeighting.jl is not used and was deprecated in favor of GeoEstimation.jl in 2020.

Solution : InverseDistanceWeighting.jl can just be removed

  1. Extremes.jl has a Manifest.toml file in the repo and conflicts with other packages

Removing the Manifest.toml from Extremes.jl as done in https://github.com/gaelforget/Extremes.jl/tree/rm_Manifest_toml should help.

For now in https://github.com/gaelforget/ClimateTools.jl/tree/update_dependencies_gf1 just removes the dependency to Extremes.jl and omits related test + code.

next steps?

The above patches allow for passing all of the ClimateTools.jl tests with updated dependencies on Julia 1.6 and 1.8

See https://github.com/gaelforget/ClimateTools.jl/actions/runs/3041967989

To me this would seem like a useful step forward given that ClimateTools.jl seems stuck at v0.23.1 from Sept 2021 and is becoming hard to use in concert with other packages.

Merging https://github.com/gaelforget/ClimateTools.jl/tree/update_dependencies_gf1 would at least allow using ClimateTools.jl and the up to date ClimateBase.jl at the same time. This could in fact help progress on the stated goal of ClimateBase.jl supporting ClimateTools.jl in the future.

With regard to Extremes.jl, I will send a PR to them and am hopeful we would just be able to bring back their next release as a dependency in ClimateTools.jl , in order to undo that part of https://github.com/gaelforget/ClimateTools.jl/tree/update_dependencies_gf1

Load function for weather stations

We need a function that will load weather stations data.

The problem is that there is no standard for such data.

Objectives

  • Canadian weather stations
  • NCEI weather stations (see NCEI package)
  • A hardcoded way to import csv data (columns is station, rows are time)

Use PermutedDimsArray

use PermutedDimsArray so that time is the first dimension. PermutedDimsArray creates a view, so that no copying is done.

Will need to test if this is accelerating calculations similarly to a real permuted array with time as the 1st dimension.

CF-Compliance for NetCDFs

Running into some issues loading data, issues arise with definitions of CF-Compliance: https://github.com/Balinus/ClimateTools.jl/blob/653a6275060eeaf988815b024f04572da07ef6e8/src/extract.jl#L31

The "project" convention is typical for most of the grids derived from models, but this isn't part of the CF standards (http://cfconventions.org/Data/cf-conventions/cf-conventions-1.7/cf-conventions.html; abridged version: https://daac.ornl.gov/submit/netcdfrequirements/). This is particularly important for analyzing reanalysis and raw data that doesn't typically have a "project" defined.

Perhaps including a warning when the files break CF-Compliance would be good? From my understanding the only global attribute fields required are "Conventions" (with capital 'C'), "institution", "references", "title", "source", and sometimes "climatology_bounds" for normal calculations.

Splitting the package?

Wondering if splitting the package would be a good idea: ClimateTools and ClimateMaps? There's a lot of dependencies problem caused by the mapping features.

GEV bias correction

Add a mixed mode bias correction, mixing POT approach with the "standard" quantile quantile mapping.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.