enram / globam-dmp Goto Github PK

@adokter is this statement correct:

Or do US pvol file need preprocessing before they can be read by biorad?

See https://travis-ci.org/enram/globam-dmp

Define which columns to include in a tabular format for vpts data (vertical profile time series). This file format will be used to generate monthly time series per radar, which are more convenient to use then individual hdf5 vp files. For discussion, see adokter/vol2bird#120

"Climate data" was listed as an external data source we plan to use for GloBAM, which means we need to document it in our DMP.

• What WP is going to use "climate data"?
• What source will be used for this data?
• Who can answer these questions? 🤷‍♂️

@adokter, if I understood correctly, vp data are being generated for the 25 year pvol archive in the US. How far along in this process are you and where are these data planned to be stored?

Ideally we can combine these with the European data on S3 (either as one or two buckets) and provide single point access via http://enram.github.io/data-repository/ ?

"Artificial light data" was listed as an external data source we plan to use for GloBAM, which means we need to document it in our DMP.

• What WP is going to use "artificial light data"?
• What source will be used for this data?
• Who can answer these questions? 🤷‍♂️

For a limited set of countries and time periods, partners of the globam project already store pvol data at the moment. It is important to have an idea of what we would have available for the envisioned period 2016-2018 (3 years of data).

On this gsheet, the third sheet available pvol data provides an overview of the existing radars. Please, provide as a comment to the specific cell the number of days you have pvol data available. We prepared columns on a year basis (2016 other partners,....) for each radar (we'll compile the information based on these comments).

@jshamoun I do think uva in particular has some data stored, so some columns are already setup for you with write access ;-)

Available here: https://github.com/adokter/vol2bird/wiki/ODIM-bird-profile-format-specification

It hasn't been discussed yet, but the DMP should probably include a section on where we will publish software and how open source these will be. Rmd should be called 8_software.Rmd.

• biorad
• processing pipeline
• analysis scripts

To build a 2-3 year archive of European pvol data for GloBAM, we need to know the temporal and geographical scope of the BALTRAD archive (requires #15). This overview will be gathered in this spreadsheet.

The Rmd will download the opera database into this repo:

However, when the bookdown is running on Travis, that file will be downloaded, but not committed. We therefore need to update .travis.yml to include those commit steps.

• radX_yyyymmddThhmmss_vradh: screenshot_ppi_VRADH 0.5 degrees
• radX_yyyymmddThhmmss_dbzh: screenshot ppi DBZH 0.5 degrees
• radX_yyyymmddThhmmss_vradh_vcp2: screenshot_ppi_VRADH 0.5 degrees, if PRF different
• radX_yyyymmddThhmmss_dbzh_vcp2: screenshot ppi DBZH 0.5 degrees, if PRF different

@CeciliaNilsson709 @adokter how do you assess the quality of pvol files for the US (gaps in coverage, etc.)? Or is this done by NOAA?

It hasn't been discussed yet, but the DMP should probably include a section on where we will publish papers and how open access these will be. Rmd should be called 8_publications.Rmd.

To get an idea of the scope of the pvol data on BALTRAD, we need a file listing. That would allow us to create an overview of the coverage.

Results from an earlier questionnaire of the individual meteo services documented here:
https://github.com/adokter/vol2bird/wiki/Radar-data-requirements-and-availability-for-ENRAM

Documenting the DMP in the wiki was easy to setup, but I wasn't completely satisfied because it does not allow 1) pull requests for suggesting changes, 2) a pdf download of the DMP (likely to be requested by funder) and 3) overviews generated by code.

I have therefore setup bookdown for this repository. The DMP can now be maintained as .Rmd files in src. Each time a commit is made to master, these will be bundled automatically by travis and generated as a website at https://enram.github.io/globam-dmp/

Advantages:

• Contributions to the DMP can be made as pull requests
• Code chunks can be included (e.g. here)
• Edits can still be made directly via GitHub (no RStudio required)
• An up-to-date pdf version of the dmp is generated: https://enram.github.io/globam-dmp/globam-dmp.pdf
• The layout of the DMP is a bit more professional than the GitHub wiki and can be further customized if need be.

Let me know if you have comments/suggestions!

I still need to:

• Transfer remaining info from wiki to .Rmd (already done for Europe)
• Close wiki
• Document how to make contributions

All time figure

To have a better idea of the pvol data coverage in BALTRAD and compare radars over years, create heatmap figure with:

• rows: radars (164)
• columns: weeks (8 years * 52 = 416 and growing)
• values: colour gradient for % of expected data files (depends on sampling interval)

Year figures

For more in depth inspection (but still ability to compare radars), create heatmap figure with:

• rows: radars (164)
• columns: days (366)
• values: colour gradient for % of expected data files (depends on sampling interval)

For both figures, we'll have to see if we should switch rows and columns. Figures could be created as RShiny and/or static.

"Weather data" was listed as an external data source we plan to use for GloBAM, which means we need to document it in our DMP.

• What WP is going to use "weather data"? Is it wind data only?
• What source will be used for this data?
• Who can answer these questions? 🤷‍♂️

Is baltrad running the latest vol2bird algorithm?

1. Download recent file: https://lw-enram.s3-eu-west-1.amazonaws.com/be/wid/2019/04/11/06/bewid_vp_20190411T064000Z_0x5.h5
2. Read with bioRad:

library(bioRad)
read_vpfiles("my_data/bewid_vp_20190411T062000Z_0x5.h5")

3. generated by: vol2bird 0.3.20

That seems to be the latest release @adokter?

Forgot about one more external data source:

"Land use" was listed as an external data source we plan to use for GloBAM, which means we need to document it in our DMP.

• What WP is going to use "land use data"?
• What source will be used for this data?
• Who can answer these questions? 🤷‍♂️

"Wind energy installations" was listed as an external data source we plan to use for GloBAM, which means we need to document it in our DMP.

• What WP is going to use data on "wind energy installations"?
• What source will be used for this data?
• Who can answer these questions? 🤷‍♂️

@adokter or @CeciliaNilsson709 could you document the US pvol data for the DMP in this document?

For reference, here's what I wrote for the European pvol data: Rmd and webpage.

From what I know, it's:

• Source: NEXRAD, but what is the access point
• License: free to use (url for this info)
• Format: odim?
• Geographical scope: any good page with a list of all NEXRAD radars? How many of these generate data? See the European section
• Temporal scope: 25 years, from x to x