aaronspring / remote_climate_data

a collection of remote climate data accessed via intake cached to disk

remote_climate_data

a collection of remote climate data accessed via intake cached to disk

Usage

import intake
cat = intake.open_catalog('https://raw.githubusercontent.com/aaronspring/remote_climate_data/master/master.yaml')
cat.atmosphere.HadCRUT5.to_dask()

To explore the whole catalog, you can try:

cat.walk()

Goal

Make data access for climate data easy:

• cacheable data
• documentation attached in metadata
• shareable catalogs
• quick vizualisations

Contribute and extend

• PRs for new remote climate datasets or useful geoshapes are very welcome

Relies on

• intake for catalogs and csv and zarr
• fsspec for caching
• intake_xarray for:
  • nc using netcdf4 [example]
  • tif using rioxarray [example]
  • grib using cfgrib [example missing]
• intake_thredds for using intake_xarray via THREDDS [example]
• intake_excel for Excel xls and xlsx [example]
• intake_geopandas for shapefiles shp [example], GeoJSON geo.json [example], GeoParquet parquet, PostGIS databases, Spatialite databases
• regionmask for aggregating over geoshapes
• hvplot for plotting [example]

Similar projects

• Pangeo's cloud data catalogs for multi GB and TB datasets: https://github.com/pangeo-data/pangeo-datastore

remote_climate_data's Issues

update caching defaults

• to_dask() not lazy when simplecache in url: intake/intake-xarray#73
• dont set default location?

add forecasts

https://www.psl.noaa.gov/forecasts/sstlim/

pd.read_csv('https://www.psl.noaa.gov/forecasts/sstlim/n4gl.frcstlead3',sep='\t',skiprows=3, names=['Verif Date','Pred-1sigma','Prediction','Pred+1sigma'])

and try with climpred

test all parameter combinations

currently only defaults are tested

cat.atmosphere.GISTEMP(t_res='ltm').describe()
'user_parameters': [{'name': 's_res',
   'description': 'spatial resolution',
   'type': 'str',
   'allowed': ['1200km', '250km'],
   'default': '1200km'},
  {'name': 'realm',
   'description': 'data region including or excluding ocean',
   'type': 'str',
   'allowed': ['landonly', 'combined'],
   'default': 'combined'},```

setup CI

• test all nc files incl. parameters with xr.open_dataset(url, chunks={}) whether available while not downloading which would take too long probably

cmorize

with derived datasets https://github.com/intake/intake/blob/master/docs/source/transforms.rst

Specify why data access works

the access model is not very clear.

• nc files via opendap works only without caching

• nc files via other sources: from http(s), ftp only works with/because of caching

• http can also work with #bytes

to do: explain better in readme

RCP and SSPs

http://www.pik-potsdam.de/~mmalte/rcps/

SSPs require login https://tntcat.iiasa.ac.at/RcpDb/dsd?Action=htmlpage&page=download

via intake-excel

air-sea CO2 fluxes

https://zenodo.org/record/5148460

SEDAC: world population

https://sedac.ciesin.columbia.edu/data/collection/gpw-v4/sets/browse

https://gist.github.com/aaronspring/9087af6864d5e8c3c7df3123f46143c8

no caching for testing purposes

maybe via parameter cache: default simplecache::

thredds server allows from different sources: opendap, httpsource and intake-thredds

https://psl.noaa.gov/thredds/catalog/Datasets/gistemp/combined/1200km/catalog.html?dataset=Datasets/gistemp/combined/1200km/air.2x2.1200.mon.ltm.comb.nc

showcase
explain differences

http and opendap changable via url parameter, but opendap no simplecache

fix quickplots: dynamic, coastlines=True, try projection

https://hvplot.holoviz.org/

upgrade binder

add:

• jupyter labextension install @jupyter-widgets/jupyterlab-manager
• jupyter labextension install @pyviz/jupyterlab_pyviz
• https://github.com/jupyterlab/jupyterlab-toc
• add import hvplot.xarray
• intake.gui.add('master.yaml')
• https://stackoverflow.com/questions/57343134/jupyter-notebooks-not-displaying-progress-bars
• postBuild file

add pre-commit and linting

add `tif` example with `rioxarray`

• data: https://zenodo.org/record/4558732
• paper: doi:10.1111/gcb.16060
• add rioxarray to env and use engine='rasterio'

cache all locally convenience function

Region: Köppen climate zones

https://crc806db.uni-koeln.de/dataset/show/kppen-geiger-classification-of-mpi-esm-p-lgm-simulation/

http://koeppen-geiger.vu-wien.ac.at/

add `cfgrib` example

• see https://github.com/NCAR/intake-thredds/blob/main/docs/source/tutorial.ipynb
• add cfgrib to env

use data hosted on `zenodo`

as #83

Include more features from naturalearthdata

https://www.naturalearthdata.com/features/

Urban areas

setup binder

Integrate intake-thredds sources

After intake/intake-thredds#6 is resolved

obs4MIP catalog

It is annoying to download obs4mip datasets. here, we could create a catalog with common observational data products. Example:

# atmos xco2 crdp: CO2 monthly satellite from 2003 to 2014
# httpServer after search on https://esgf-data.dkrz.de/search/obs4mips-dkrz/
url='https://esgf-data1.ceda.ac.uk/thredds/fileServer/esg_esacci/ghg/data/obs4mips/crdp_3/CO2/v100/xco2_ghgcci_l3_v100_200301_201412.nc'
import intake_xarray
intake_xarray.NetCDFSource('simplecache::'+url,storage_options=dict(simplecache={'same_names':True, 'cache_storage':'my_cache'})).to_dask()

# GPCP 1.3 GB
url='https://dpesgf03.nccs.nasa.gov/thredds/fileServer/obs4MIPs/observations/NASA-GSFC/Obs-GPCP/GPCP/1DD_v1.2/atmos/pr_GPCP-1DD_L3_v1.2_19961001-20110630.nc'
ds = intake_xarray.NetCDFSource('simplecache::'+url,storage_options=dict(simplecache={'same_names':True, 'cache_storage':'my_cache'})).to_dask()

How to:

1. Search for data product: https://esgf-data.dkrz.de/search/obs4mips-dkrz/
2. select HTTPServer
3. get link
4. create catalog entry

Advantages compared to downloading yourself:

• collection of links
• obs4mip is cmorized in units and variable names
• automatic caching enabled

Open questions:

• How to organize the catalog? realm:name:freq as nested catalogs?

Population density example rio

https://aetperf.github.io/2022/03/18/Plotting-population-density-with-datashader.html

binder fails

atm co2 gridded

https://www.iup.uni-bremen.de/carbon_ghg/data/Satellite_XCO2_OBS4MIPS_v4p2_2x2_4C_TN_v1.pdf

https://www.iup.uni-bremen.de/carbon_ghg/data/xco2_c3s_l3_v42_200301_201912_2x2.nc

Opendap sources

https://earthdata.nasa.gov/collaborate/open-data-services-and-software/api/opendap/opendap-servers/

Wishlist for new datasets

• pattern= catalog : dataset name : url : comment

• ocean: World Ocean Atlas: https://www.nodc.noaa.gov/OC5/woa18/ : different versions and variables via parameter #15

• global carbon budget with https://github.com/edjdavid/intake-excel #22

• land: precipitation: https://psl.noaa.gov/data/gridded/tables/precipitation.html:

• Mauna Loa CO2 netcdf ftp://aftp.cmdl.noaa.gov/data/trace_gases/co2/in-situ/surface/mlo/ #19

• climate indices: see http://iridl.ldeo.columbia.edu/SOURCES/.Indices

  • AMO: https://climatedataguide.ucar.edu/sites/default/files/AMO_HADLEY.1870-2010.CLM_1901-1970.nc
  • Nino 1,2,34,3,4 #29

• reanalysis #48

Resources:

• https://www.psl.noaa.gov/data/gridded/index.html
• http://iridl.ldeo.columbia.edu/SOURCES/
• http://data.ceda.ac.uk/
• http://dap.nci.org.au/thredds/catalog.html
• https://psl.noaa.gov/thredds/catalog/Datasets/catalog.html
• https://coastwatch.pfeg.noaa.gov/erddap/griddap/index.html?page=1&itemsPerPage=1000
• https://climate.metoffice.cloud/dashboard.html

Allow for Constraint Expression Syntax when using opendap

https://earthdata.nasa.gov/collaborate/open-data-services-and-software/api/opendap/opendap-user-guide

Why: allows to subselect data before downloading

Also for averaging/dods

naturalearthdata fails

website seems down, firefox shows security issues

Regen

http://dapds00.nci.org.au/thredds/catalog/ks32/CLEX_Data/REGEN_AllStns/v1-2019/catalog.html

https://hess.copernicus.org/articles/24/919/2020/

Standard temp datasets

As listed in https://www.carbonbrief.org/analysis-why-the-new-met-office-temperature-record-shows-faster-warming-since-1970s

Add more carbon

https://www.bgc-jena.mpg.de/SOCOM/

add yamllint

https://github.com/adrienverge/yamllint/blob/4f1bbc33dcb550b27a77e10c9831b9ff732bc5a8/docs/integration.rst

Formatting multilines

description: |
text
Text

lists

update contribution guide

• add doi.org to resolve link, ensure done
• explain sample entry