Cumulative death predictions below current levels about covid19-forecast-hub

[nickreich]

my understanding is that they don't calibrate directly on death data, rather on case data, so I don't think the alignment is off. I'll flag @jbracher for a double-check of dates, but I'm not surprised by this. For all the flack people give to IHME, it's actually surprisingly hard to stand up a good, new, not buggy model quickly.

[jbracher]

@salauer If I understand things correctly they are currently not included in the ensemble (which is only IHME + LANL), so problems could arise as soon as they are.

[nickreich]

From CU folks: "the April 16th projections are the first that explicitly use observed deaths for model optimization - this will continue going forward. Earlier projections used only case data." So I hope this issue will be solved with the latest round of forecast data.

[nickreich]

is this a bug or just bad models?

[salauer]

It could be bad models, but it would be good to rule out other possibilities.

Does it have to do with when the predictions were made? Maybe the predictions were made at a time that doesn't align with the weeks used in the display.

Are the prediction times off by a week?

It appears that these are mostly for the Columbia models, so maybe something isn't aligned correctly with them.

[jbracher]

Hi @salauer, thanks for spotting this, I was not aware of this problem. I just checked the visualization online against the raw data and I don't think it is due to the re-formatting. But I'd be thankful if somebody took the time to take a quick glance at my check and see if there is some obvious misunderstanding (pasted below).

Is this something we should check in the "plausibility checks"? In order to at least know if a model shows such behaviour?

# get info on timing of 1 wk ahead forecasts
templ <- read.csv("template/covid19-death-forecast-dates.csv", stringsAsFactors = FALSE)
templ$timezero <- as.Date(templ$timezero)
templ$forecast_1_wk_ahead_end <- as.Date(templ$forecast_1_wk_ahead_end)

# what is the end date for 1 wk ahead forecast issued at 2020-04-13?
timezero <- as.Date("2020-04-13")
forecast_1_wk_ahead_end <- templ[templ$timezero == timezero, "forecast_1_wk_ahead_end"]
# "2020-04-18"

# check what we get for Georgia for this date (cumulative deaths, using raw data)

# CU60-contact
dat60 <- read.csv("data-raw/CU/Projection_April12/cdchosp/state_cdchosp_60contact.csv")
dat60$Date <- as.Date(dat60$Date, format = "%m/%d/%y")
dat60[dat60$Date == forecast_1_wk_ahead_end & dat60$location == "Georgia",
      c("location", "Date", "cdeath_50")]
#         location       Date cdeath_50
# 316  Georgia 2020-04-18       286

# CU70-contact
dat70 <- read.csv("data-raw/CU/Projection_April12/cdchosp/state_cdchosp_70contact.csv")
dat70$Date <- as.Date(dat70$Date, format = "%m/%d/%y")
dat70[dat70$Date == forecast_1_wk_ahead_end & dat70$location == "Georgia",
      c("location", "Date", "cdeath_50")]
#        location       Date cdeath_50
# 316  Georgia 2020-04-18       327

# CU80-contact
dat80 <- read.csv("data-raw/CU/Projection_April12/cdchosp/state_cdchosp_80contact.csv")
dat80$Date <- as.Date(dat60$Date, format = "%m/%d/%y")
dat80[dat80$Date == forecast_1_wk_ahead_end & dat80$location == "Georgia",
      c("location", "Date", "cdeath_50")]
#              location       Date cdeath_50
# 316  Georgia 2020-04-18       385

# these all agree with what is displayed in the interactive plot online, but are below
# the cumulative deaths already observed in the previous week:

truth <- read.csv("data-processed/truth-cum-death.csv")
truth$date <- as.Date(truth$date, format = "%m/%d/%y")
subset(truth, date == (forecast_1_wk_ahead_end - 7) & location_name == "Georgia")
#               date location location_name value
# 4171 2020-04-11       13       Georgia   429

[salauer]

Thanks @jbracher, it does appear that the models are making poor predictions as opposed to any technical issue. It's interesting (and good?) that these predictions don't appear to bring the ensemble predictions below that threshold. Though I am curious as to why that is.