reichlab / covid19-forecast-hub Goto Github PK

I suggest we reorganize the data so that forecasts are separate from truth, e.g.

data-raw/forecasts
data-raw/truth
data-processed/forecasts
data-processed/truth

The subdirectory structure within the forecasts/ subdirectories would be the same as it is now.

Also, perhaps we should include nytimes "gold-standard" data in addition to the JHU data.

I already pushed my code, but it should only be run & results committed once the April 12 forecasts are available. Currently I use the April 9 forecasts to test the code, but these are not actually going to be used.

• Instructions on how to check file locally
• Summary of what checks are currently in place

Some additional validations

• ensure that we are checking for all required column names as required by the repo (right now we are requiring forecast_date and target_end_date which are not part of Zoltar) can we require these?
• are we validating the FIPS locations based on the specific set of valid numbers, or just any string of a number between 01 and 95? I would prefer the former, so we are doing it specifically for accepted FIPS.
• can we institute a more complex check to ensure that people are aligning forecast_date and target_end_date correctly? I will explain more below.
• Require point estimates (exactly one point estimate per location/target tuple) - we know from Katie's code that the forecast_date column is the same for the entire file (based on filename)
• update https://github.com/reichlab/covid19-forecast-hub/wiki/Validation-Checks

I'm working on this.

Currently most of the code is in the code/ directory and recently organized into subdirectories. As a general principle, I suggest we move code closer to the data it is used on. For example, I suggest we move raw data processing scripts to the data-raw/ folder.

The code/ directory could still be used for functions (rather than scripts) that are used in multiple scripts.

@tomcm39

https://github.com/reichlab/covid19-forecast-hub/blob/master/data-processed/UMass-ExpertCrowd/2020-04-13-UMass-ExpertCrowd.csv

It needs point estimates to be in this format:

2020-04-12,1 day ahead cum death,2020-04-13,31,Nebraska,point,NA,6

This newly added file does not have required fields: forecast_date and target_end_date.

For consistency,

Pull request incoming.

In addition to the missing fields in #66, the newest MOBS processed data file has a filename that is non-standard (for me) and is causing me issues with reading processed data for the shiny data-processing app.

Although I could update the data reading script, I think the real issue is that we don't seem to have a standardized filename processed data. I was assuming "-" was a reserved character such that the files are named

YYYY-MM-DD-team-model.csv

Can we set this as a standard?

The following files need required fields forecast_date and target_end_date:

data-processed/CU-60contact/2020-04-13-CU-60contact.csv
data-processed/CU-70contact/2020-04-13-CU-70contact.csv
data-processed/CU-80contact/2020-04-13-CU-80contact.csv
data-processed/CU-nointerv/2020-04-13-CU-nointerv.csv

showing JHU IDD has 50 states but it actually has 0?

Could we use the drop-down menu that we have previously used for "season" to toggle between different forecast targets, e.g. incident deaths, cumulative deaths, hospitalizations, etc...? how much customization would this take?

• specify preferred quantiles
• specify what scoring will be done
• specify when death data will be retrieved for scoring

what are the next-phase targets that we want to include? likely we should phase these in slowly, to reduce strain on creating checks, visualizations, ensembles, for new targets. candidates are:

• incident hospitalization demand by week/day?
• ICU bed demand by week/day?
  ...

There are two competing priorities here:
(1) record all (or nearly all - do we really want to store every update, even if daily?) forecasts made by teams, as they make them [useful for "tracker"-like sites that want all versions and real-time updates]
(2) record forecasts made by teams that are available at a specific time, and use them to build an ensemble. realistically, for the foreseeable future we might just want to update the ensemble once a week. [useful for our standardizing our ensemble]

Here is one proposal for how to do this:

• we have the data-processed directory contain all (or nearly all) forecasts from each team. no restrictions on when these forecasts are submitted.
• each file is marked with the date the forecast was made. This would change a bit our restriction right now that these YYYY-MM-DD's only refer to Mondays. I'm going to refer to this date in the filename as fcast_date below.
• we set really clear guidelines for when "1 wk ahead" means epiweek(fcast_date) and when it means epiweek(fcast_date)+1. for example, we say that if weekday(fcast_date) is Thursday, Friday or Saturday, then "1 wk ahead" means epiweek(fcast_date)+1 and otherwise epiweek(fcast_date). (I don't feel that strongly about where the threshold is for switching over. Could be Tuesday, could be Thursday.)
• to reinforce this and avoid inadvertent errors in assignment of targets to days/weeks, we could also accept a new column name in the files that would be end_date, so files submitted with fcast_date of 2020-04-23 (thursday of EW 17) or 2020-04-27 (Monday of EW 18) would both have a "1 wk ahead" forecast with end_date of 2020-05-02 (Saturday of EW 18).
• on Mondays at a fixed time (6pm ET?) we run an ensemble script that finds all available forecasts from a team made since the preceding Thursday (i.e. 4 days prior) and takes the most recent forecast to include in the ensemble.

@jarad I see up to 41 day ahead targets in the recent 2020-04-26 CU files, but only max_n=9 for the day_ahead targets in the app.

Ex. Guam

All team forecasts should be in a subdirectory of data-raw/, but these files

https://github.com/reichlab/covid19-forecast-hub/blob/master/data-raw/2020-04-19-Imperial-ensemble1.csv
https://github.com/reichlab/covid19-forecast-hub/blob/master/data-raw/2020-04-19-Imperial-ensemble2.csv

are directly in the data-raw/ folder.

I would create a pull request, but these files have differences to the files in the data-raw/Imperial subdirectory, so I'm not sure which versions should be preserved.

@youyanggu could you add a metadata file to the YYG-ParamSearch Model?

See https://github.com/reichlab/covid19-forecast-hub/blob/master/data-processed/UMass-ExpertCrowd/metadata-UMass-ExpertCrowd.txt for an example. We need the metadata in order to visualize the model. Thanks!

This is related to #66 and #67. I believe

https://github.com/reichlab/covid19-forecast-hub/blob/master/data-processed/MOBS_NEU-GLEAM_COVID/2020-04-27-MOBS_NEU-GLEAM-COVID-19_v1.csv

should be in

https://github.com/reichlab/covid19-forecast-hub/tree/master/data-raw

This file includes rownames in the first column which is non-standard.

Also, the file does not include target_end_date.

In processed data, the location is "US", but the location_name can be "US", "United States", or . Specifically, it is "United States" in UTexas data and in Imperial data.

Add a subsection enumerating teams / sources of forecasts we are planning to include + links to their repositories or websites

As of 2020-04-26 there are global files, with country-level forecasts from LANL, e.g.
https://covid-19.bsvgateway.org/forecast/global/files/2020-04-26/confirmed/2020-04-26_confirmed_quantiles_global_website.csv

These files should be included in the raw data download script for LANL.

@tomcm39

Georgia, Indiana, Alabama, Arkansas, and Iowa to name a few examples

Write a script that does some plausibility checks for cleaned data, eg:

• no quantile crossing
• quantiles for cumulative deaths greater or equal than those for incident
• quantiles for cumulative deaths non-decreasing over time
• cumulative week-ahead and corresponding day-ahead forecasts coincide
  Maybe related to #13 ?