add additional validations about covid19-forecast-hub

[nickreich]

@matthewcornell I actually see these as essential checks for the data integrity, although I recognize that they are not essential for Zoltar. Does it not make sense to add them in zoltpy? I was thinking that is where it makes sense for them, but am open to other places (or having some other python scripts run that aren't in zoltpy) as well.

[nickreich]

For x day ahead targets the target_end_date should be forecast_date + x

[nickreich]

Assume a numbering of weekdays where Sunday=1, Monday=2,..., Saturday=7.

For x week ahead targets:
weekday(target_end_date) should be 7 (i.e., a Saturday) AND

if weekday(forecast_date)<=2
target_end_date should = forecast_date + (weekday(target_end_date) - weekday(forecast_date)) + 7*(x-1)
% this ensures that the 1-week ahead forecast is for the next saturday, and so on

if weekday(forecast_date)>2
target_end_date should = forecast_date + (weekday(target_end_date) - weekday(forecast_date)) + 7*x
% this ensures that the 1-week ahead forecast is for the saturday after next, and so on

[matthewcornell]

I documented the checks that are done via zoltpy here: https://github.com/reichlab/covid19-forecast-hub/wiki/Validation-Checks . B/c it only checks the essential columns, I assume the above checks should be done in R (i.e., they are "statistical sanity checks," not "structural" ones).

[matthewcornell]

• Does it not make sense to add them in zoltpy? It's a judgement call. From a functional perspective, no b/c they have nothing to do with what gets sent to zoltar. But if the checks depend not just on row-level validation but on prediction-level (grouped) quantiles, then that could be a stronger argument for putting them in zoltpy, which already groups them for zoltar. However, to me it looks like they are all row-level ones.
• are we validating the FIPS locations based on the specific set of valid numbers: no. I can add that.

Reviewing, the only COVID-specific code in zoltpy is: 1) location being a FIPS code, and 2) the target names. I can refactor the code to extract these as args. This would make me feel a better about putting more validation in zoltpy.

[katiehouse3]

• Could we add another check to require point estimates?

If the forecast does not have point estimates, it breaks the visualization because it has no point to plot. Unless @nickreich we would like to be flexible in allowing point estimates?

[jarad]

We probably need to split this issue up into multiple issues.

I suggest we eliminate forecast_date and target_end_date. We should use the date in the filename as the forecast_date and calculate the target_end_date using the formula above. This reduces duplication and therefore the possibilities for errors. Although I understand it eliminates the ability to make checks.

In my script to read in the processed data, I check explicitly for the 50 state fips codes using the fips file.

@matthewcornell mentioned checks in R, where are these checks?

[matthewcornell]

@jarad thanks for the pointer to the file. Re: eliminate forecast_date and target_end_date, I'll leave that to you all, but with the setup like it is, I think changing something this major will meet resistance. re: checks in R, I don't recall this. I haven't ported the quantile code from zoltpy to zoltr yet.

[matthewcornell]

@katiehouse3 is the requirement exactly one point estimate per location/target pair, or are multiple ones allowed?