This repository contains the instructions and examples for using the ENS-10 dataset.
Abstract: Post-processing ensemble prediction systems can improve weather forecasting, especially for extreme event prediction. In recent years, different machine learning models have been developed to improve the quality of the post-processing step. However, these models heavily rely on the data and generating such ensemble members requires multiple runs of numerical weather prediction models, at high computational cost. This paper introduces the ENS-10 dataset, consisting of ten ensemble members spread over 20 years (1998--2017). The ensemble members are generated by perturbing numerical weather simulations to capture the chaotic behavior of the Earth. To represent the three-dimensional state of the atmosphere, ENS-10 provides the most relevant atmospheric variables in 11 distinct pressure levels as well as the surface at 0.5-degree resolution. The dataset targets the prediction correction task at 48-hour lead time, which is essentially improving the forecast quality by removing the biases of the ensemble members. To this end, ENS-10 provides the weather variables for forecast lead times T=0, 24, and 48 hours (two data points per week). We provide a set of baselines for this task on ENS-10 and compare their performance in correcting the prediction of different weather variables. We also assess our baselines for predicting extreme events using our dataset. The ENS-10 dataset is available under the Creative Commons Attribution 4.0 International (CC BY 4.0) licence.
For any questions, please create an issue.
To install requirements:
pip install -r requirements.txt
CliMetLab is a Python package to simplify accessing meteorological and climate datasets. ENS-10 can be downloaded using a CliMetLab Plugin.
For a fixed date, the data points can be accessed for both surface-level data, and pressure-level data for above-ground forecasts using a few lines of code:
!pip install climetlab climetlab-maelstrom-ens10
import climetlab as cml
# Pressure-level data
ds = cml.load_dataset("maelstrom-ens10", date='20170226', dtype='pl')
# Surface-level data
ds = cml.load_dataset("maelstrom-ens10", date='20170226', dtype='sfc')
# Alternatively, the year can be omitted, and pressure levels are given by default:
# ds = cml.load_dataset("maelstrom-ens10", date='0226')
# Convert dataset to xarray data
ds.to_xarray()
For demo notebooks, see here.
The dataset is hosted on the ECMWF servers. All files can be downloaded alternatively using this link. All files have 2018
prefix.
To train the baseline model(s) in the paper, run this command:
python Train.py --model <model_name> --data-path <path_to_data> --target-var <predicted_variable>
baselines
-- this folder contains all scripts for running the baseline models.baselines/utils
-- this folder contains all scripts for extracting the data, converting from GRIB to Numpy, and metric.EFI
-- this folder contains all scripts for extracting the extreme forecast index over the dataset.
The ENS-10 dataset is available under the Creative Commons Attribution 4.0 International (CC BY 4.0) licence (see here).
@article{ashkboos2022ens,
title={ENS-10: A Dataset For Post-Processing Ensemble Weather Forecast},
author={Ashkboos, Saleh and Huang, Langwen and Dryden, Nikoli and Ben-Nun, Tal and Dueben, Peter and Gianinazzi, Lukas and Kummer, Luca and Hoefler, Torsten},
journal={arXiv preprint arXiv:2206.14786},
year={2022}
}