earthobservations / gribmagic

Generic weather forecast downloader

GribMagic - generic weather forecast downloader

Simulations are believed by no one except those who conducted them.

Experimental results are believed by everyone except those who conducted them.

ANONYMOUS

About

The goal of the GribMagic project is to unify the download process of public GRIB1/GRIB2 and netCDF data from numerical weather prediction models originating from different organizations and data providers.

This is a work in progress, as such GribMagic is currently considered to be beta software. As this is an early-stage project, contributions are highly appreciated.

Details

The software package includes different code bases. Both provide their features by means of a gribmagic subcommand.

• GribMagic Unity program: gribmagic unity --help
• DWD GRIB Downloader program: gribmagic dwd --help

You can find corresponding invocation examples later in this document.

Setup

Install the ecCodes package by ECMWF.

# Debian Linux
apt-get install --yes libeccodes0 cdo

# macOS/Homebrew
brew install eccodes cdo

Install GribMagic Python package.

pip install gribmagic --upgrade

Run GribMagic Unity program

Ad hoc usage

# List labels of available models.
gribmagic unity list

# Acquire data.
gribmagic unity acquire --model=dwd-icon-eu --timestamp=2021-10-03T00:00:00Z --target=.gribmagic-data

Configuration

To optionally adjust configuration settings, you can define the following environment variables:

GM_DATA_PATH={PATH_TO_PROJECT}/data

GM_MODEL_CONFIG={PATH_TO_PROJECT}/config/model_config.yml"
GM_MODEL_VARIABLES_MAPPING={PATH_TO_PROJECT}/config/model_variables_mapping.yml"
GM_MODEL_VARIABLES_LEVELS_MAPPING={PATH_TO_PROJECT}/config/model_variables_levels_mapping.yml"

ECCODES_DEFINITION_PATH=/usr/share/eccodes/definitions:/usr/local/opt/eccodes/share/eccodes/definitions

Run DWD GRIB Downloader program

# Install DWD GRIB Downloader program.
gribmagic install dwd-grib-downloader

# Acquire wind-specific parameters from ICON-D2.
wget https://raw.githubusercontent.com/earthobservations/gribmagic/98da3fd4f/examples/dwd/recipe_d2_wind.py
gribmagic dwd acquire --recipe=recipe_d2_wind.py --timestamp="2021101800" --output=.gribmagic-data/raw

# Acquire assorted parameters from ICON-GLOBAL.
wget https://raw.githubusercontent.com/earthobservations/gribmagic/98da3fd4f/examples/dwd/recipe_global_assorted.py
gribmagic dwd acquire --recipe=recipe_global_assorted.py --timestamp="2021101800" --output=.gribmagic-data/raw

When omitting the --timestamp parameter, the most recent modelrun is automatically selected. When omitting the --output parameter, it can be supplied using the GM_DATA_PATH environment variable.

In this manner, the most compact form to invoke gribmagic dwd would be something like:

export GM_DATA_PATH=.gribmagic-data/raw
gribmagic dwd acquire --recipe=recipe_d2_wind.py

Run bbox tool

Extract area of interest from GRIB files using a bounding box.

Extract subset by coordinates, a space-separated list of lat_min lat_max lon_min lon_max.

gribmagic smith bbox \
  ".gribmagic-data/raw/icon-d2/**/*regular-lat-lon*.grib2" \
  --output=.gribmagic-data/subgrid/icon-d2 \
  --bbox=46.0 47.5 14.5 16.8

Extract subset by 2-letter country name.

gribmagic smith bbox \
  ".gribmagic-data/raw/icon-d2/**/*regular-lat-lon*.grib2" \
  --output=.gribmagic-data/subgrid/icon-d2 \
  --country=AT

For plotting the result on a map, install Magics:

# Linux
apt-get install --yes libmagplus3v5

# macOS
make magics-install
make magics-info
export MAGPLUS_HOME=/usr/local/opt/magics-4.9.3

# Install GribMagic with Magics bindings
pip install gribmagic[plotting]

and use the --plot option:

gribmagic smith bbox \
  ".gribmagic-data/raw/icon-d2/**/*regular-lat-lon*.grib2" \
  --output=.gribmagic-data/subgrid \
  --country=AT \
  --plot

Run regridding tool

Acquire grid information files.

gribmagic install dwd-grid-information

Process regridding on a single file. The default is to assume ICON GLOBAL files and the regridding will take place with 0.250° resolution.

gribmagic smith regrid \
  ".gribmagic-data/raw/icon/**/*icon_global_icosahedral*.grib2" \
  --output=.gribmagic-data/regridded/icon-global

When aiming to regrid with 0.125° resolution, please specify the --resolution=0.125 argument, like:

gribmagic smith regrid \
  ".gribmagic-data/raw/icon/**/*icon_global_icosahedral*.grib2" \
  --output=.gribmagic-data/regridded/icon-global \
  --resolution=0.125

Run program in Docker

To use GribMagic in a Docker container, you have to build the Docker image like

docker build --tag gribmagic .

and then invoke it like

docker run -it --volume=$PWD/.gribmagic-data:/var/spool/gribmagic gribmagic:latest gribmagic unity acquire --model=dwd-icon-eu --timestamp=2021-10-03T00:00:00Z

Development

Acquire source code

git clone https://github.com/earthobservations/gribmagic
cd gribmagic

Run software tests

# Run all tests.
make test

# All tests, with code coverage report.
make test-coverage

# Run specific tests, with coverage report.
.venv/bin/pytest -vvv --cov-report=term-missing --cov=gribmagic.smith.regrid -k test_regrid

# Run tests in parallel.
make test-parallel
make test-coverage-parallel

---

Content attributions

The copyright of data sets, images and pictograms are held by their respective owners, unless otherwise noted.

Banner image

• 2 metre temperature from ECMWF, rendered using Magics.
  • Source: MARS labelling or ensemble forecast data (2013-03-25)
  • URL: http://download.ecmwf.int/test-data/magics/2m_temperature.grib

gribmagic's Issues

[unity] DWD ICON: Data for specific fields is published on multiple levels

Hi again,

while looking at #2, we found that the folders hosting parameters for temperature [1,2] and wind [3,4] not only contain model-level data, but also pressure-level data. So, we will probably have to improve model_variables_levels_mapping.yml and the respective downstream infrastructure to accept a list of labels on the right hand side of the mapping.

With kind regards,
Andreas.

[1] https://opendata.dwd.de/weather/nwp/icon/grib/00/t/
[2] https://opendata.dwd.de/weather/nwp/icon-eu/grib/00/t/
[3] https://opendata.dwd.de/weather/nwp/icon/grib/00/u/
[4] https://opendata.dwd.de/weather/nwp/icon-eu/grib/00/u/

Problems installing GribMagic

Hi. When i run the file i keep getting invalid syntax when i run python setup.py install.

Add acquisition for ICON-D2

@mhaberler suggested at #4 (comment):

Note Cosmo-D2 is on the way out and is currently being replaced by Icon-D2 which is sort-of preproduction status (but stable as far as I can judge).

Thanks. So, we should also add acquisition for ICON-D2 to the mix.

ICON-D2: Demo pipeline for acquisition, area of interest extraction and plotting

Dear Michael,

based on your improvements to the dwd-open-data-downloader by @EduardRosert and @bjoern-reetz you published at EduardRosert/docker-dwd-open-data-downloader#2 (thanks a stack to all of you!), I've just added a demo of a humble pipeline machinery [1] which might resonate with your proposal at earthobservations/wetterdienst#271 (comment).

The goal is to have as much details available as Python code as possible. So, one detail that was important to me was to have the "download/processing recipe" available as Python code in order to lay the groundwork of evolving this into different directions, see [2].

It is really only a demo, completely unpolished yet and it might not even work on Windows. It has to be invoked from the toplevel directory. A short synopsis about installation and operation would be:

# Acquire sources
git clone https://github.com/earthobservations/gribmagic
cd gribmagic

# Install ECMWF ecCodes, Climate Data Operators (CDO) and ECMWF Magics. YMMV. See also `docs/magics.rst`.
# See also https://confluence.ecmwf.int/display/ECC and https://code.mpimet.mpg.de/projects/cdo.

# Linux
apt-get install --yes libeccodes0 cdo libmagplus3v5

# macOS
brew install eccodes cdo
make magics-install
make magics-info

# Python virtualenv FTW
python3 -m venv .venv
source .venv/bin/activate

# Install Python package
pip install --editable=.[plotting]

# Install improved DWD Open Data Downloader
gribmagic install dwd-grib-downloader

# Invoke the machinery
./demo/acquire_and_bbox.sh

While still in its infancy, I believed it would be a good idea to share this with you and will be happy about any feedback.

With kind regards,
Andreas.

[1] https://github.com/earthobservations/gribmagic/blob/8d5ffdc/demo/pipeline.sh
[2] https://github.com/earthobservations/gribmagic/blob/8d5ffdc/demo/pipeline/recipe.py

cc @gutzbenj, @wetterfrosch, @mhaberler

---

Demo plot

-- .gribmagic-data/subgrid/bbox_46.0_47.5_14.5_16.8/png/icon-d2_germany_regular-lat-lon_pressure-level_2020122212_001_950_v.grib2.png

[unity] DWD ICON: Folder "aswdiff_s" missing

Hi there,

within model_variables_mapping.yml, diffuse_horizontal_irradiance is mapping to DWD ICON's aswdiff_s parameter field. However, there [1-4] are only folders called aswdifd_s and aswdifu_s.

With kind regards,
Andreas.

[1] https://opendata.dwd.de/weather/nwp/icon/grib/00/
[2] https://opendata.dwd.de/weather/nwp/icon-d2/grib/00/
[3] https://opendata.dwd.de/weather/nwp/icon-eu/grib/00/
[4] https://opendata.dwd.de/weather/nwp/icon-eu-eps/grib/00/

[unity] DWD ICON: Model level not honored when computing filenames

Hi there,

within model_variables_levels_mapping.yml, temperature is designated as a model-level field. However, the model level is missing from the filename, so GribMagic croaks like

WARNING: Access failed: 404 Client Error: Not Found for url: [...]

The computed URL looks like [1] but it should be [2] instead.

With kind regards,
Andreas.

[1] https://opendata.dwd.de/weather/nwp/icon-eu/grib/00/t/icon-eu_europe_regular-lat-lon_model-level_2020121900_000_T.grib2.bz2
[2] https://opendata.dwd.de/weather/nwp/icon-eu/grib/00/t/icon-eu_europe_regular-lat-lon_model-level_2020121900_000_10_T.grib2.bz2

DWD COSMO-D2: Data is published in multiple grid variants

Hi again,

we just found that data of DWD COSMO-D2 within the "temperature 2m" folder [1] is published with multiple grid variants like regular-lat-lon vs. rotated-lat-lon.

Parameter folders for "temperature" [2] also bring in the levels-axis like model-level vs. pressure-level as outlined within #3, so this folder and friends contains four (4) different sets of data in total.

As @mhaberler pointed out at #3 (comment), the docker-dwd-open-data-downloader by @EduardRosert also accounts for that by discriminating between "grids": ["regular-lat-lon", "rotated-lat-lon"], see models.json.

With kind regards,
Andreas.

[1] https://opendata.dwd.de/weather/nwp/cosmo-d2/grib/00/t_2m/
[2] https://opendata.dwd.de/weather/nwp/cosmo-d2/grib/00/t/