project-aero / measles-ews Goto Github PK

Need to get the random walk right for transmission in the post-MLE pomp process model. Currently, it is just being perturbed, which is not a random walk because there is no dependence on the previous value.

There was an error (now fixed, dae9b94) in the make-pomp-simulator-function.R code: I didn't reset initial values for exposed and infected classes. This doesn't make much of difference for the elimination simulations because they burn in from initial conditions, but it does impact the emergence simulations.
For emergence simulations, initial exposed and infected should be set to zero. The function now allows this. Also, simulations now start from the middle of the first year so that dynamics start in the trough of seasonal transmission dynamics. Again, not much of an impact for the elimination simulations, but this better represents the scenario for re-emergence after a big outbreak.

For consistency, need to re-run:

• simulate-emergence-grid.R (DATE COMPLETED: 2019-zz-zz)
• simulate-elimination-grid.R (DATE COMPLETED: 2019-zz-zz)
• analyze-emergence-grid-sims.R (DATE COMPLETED: 2019-zz-zz)
• analyze-elimination-grid-sims.R (DATE COMPLETED: 2019-zz-zz)

R_E is different in the SEIR model depending on the vital dynamics (demography) included. Check with Eamon to make sure these are all correct in the 2 different versions of the pomp model: the fitting model and the simulating model.

Calculate the suite of EWS on the actual data for each city. Use bandwidth of ~30.

Need to streamline the estimate-transmission-state.R code, perhaps by breaking into one script that has a function for defining the pomp model and one for looping over cities for actual filtering.

Need to add section on the demographic data interpolation to the SI material.

@e3bo, can you take a look at my script here: https://github.com/project-aero/measles-ews/blob/master/code/calculate-ews.R

I have set the bandwidth to 35 weeks and backward_looking = TRUE, but values for the stats start to show up at the 10th observation. I thought I wouldn't see anything until the 36th observation, e.g., once there are 35 data points behind the focal data point to calculate the EWS. Thanks for your thoughts.

Need to add an analysis that implements Eamon's approach for quantifying autocorrelation as outlined in the Distance to epidemic threshold paper. It should serve as the EWS for elimination, not lag-1 autocorrelation as currently presented.

Snippet of code here:

get_fit <- function(y, tstep = 1/52, est_K = FALSE, cutoff = .06) {
  x <- (seq_along(y) - 1) * tstep
  start <- list()
  im <- match(TRUE, abs(y) < cutoff)
  xs <- x[1:(im - 1)]
  ys <- y[1:(im - 1)]
  start$gamma <- try(unname(coef(lm(log(I(abs(ys))) ~ xs))["xs"]))
  if (!inherits(start$gamma, "try-error")){
    spec <- spectrum(y, plot = FALSE, na.action = na.exclude)
    start$omega <- spec$freq[which.max(spec$spec)] / tstep
    start$a <- 0
    fit_osc <- try(minpack.lm::nlsLM(
      y~sqrt(1 + a^2) * exp(x * gamma) * sin(x * omega + atan2(1, a)),
      start = start, na.action = na.exclude,
      control = minpack.lm::nls.lm.control(maxiter = 1000)))
    if (est_K) {
      fit_decay <- try(minpack.lm::nlsLM(y~K * exp(x * gamma),
                                         start = list(gamma = start$gamma, K = y[1]), na.action = na.exclude,
                                         control = minpack.lm::nls.lm.control(maxiter = 1000)))
    } else {
      K <- y[1]
      fit_decay <- try(minpack.lm::nlsLM(y~K * exp(x * gamma),
                                         start = list(gamma = start$gamma),na.action = na.exclude,
                                         control = minpack.lm::nls.lm.control(maxiter = 1000)))
    }
    if (inherits(fit_osc , "try-error")) {
      e_osc <- Inf
    } else {
      e_osc <- fit_osc$m$resid()
    }
    if (inherits(fit_decay, "try-error")) {
      e_decay <- Inf
    } else {
      e_decay <- fit_decay$m$resid()
    }
    nll <- function(resids) {
      n <- length(resids)
      (sum(resids ^ 2))
    }
    aic <- c(constant = nll(y), fit_decay = nll(e_decay) + 2 * (1 + est_K),
             fit_osc = nll(e_osc) + 2 * 3)
    fits <- list(constant = "constant_y=0", fit_decay = fit_decay, fit_osc = fit_osc)
    
    coefests <- try(coef(fits[[which.min(aic)]])[c("omega", "gamma", "a")])
    if (inherits(coefests, "try-error")){
      coefests <- c(NA, NA, NA)
    }
    names(coefests) <- c("omega", "gamma", "a")
    c(list(coef = coefests), fits)
  } else {
    c(list(coef = c("omega" = NA, "gamma" = NA, "a" = NA),
           fits = list(constant = "contant_y=0",
                       fit_decay = NA, fit_osc = NA)))
  }
}

cases <- readRDS(datafile)
cases <- cases %>%
  filter(year > 1994) %>%
  filter(region == "Maradi (City)") %>%
  pull(cases)

y <- acf(cases, lag.max = length(cases)-30, plot = TRUE)
acf_fits <- get_fit(y = as.numeric(y[[1]]))
g <- coef(acf_fits$fit_osc)["gamma"]
w <- coef(acf_fits$fit_osc)["omega"]
distance_to_threshold <- sqrt((g)^2 + (w)^2)

Take another look at the analysis of temporal correlations between EWS calculated over a moving window and the effective reproduction number. Perhaps only show for high performing EWS. Visualize as a scatter plot.

Need to make AUC figures more intuitive. Pair each (emergence and elimination) with example time series and EWS calculation to make it clear what goes into the figure. Then separate emergence and elimination.

There is a persistent filtering failure, which I've traced to the conditional likelihood for the first observation. The problem is that cases gets zeroed out at each observation, meaning cases = 0 for all particles when evaluating the first observation. It's also just better practice to estimate states right before the first observation. So, that's what needs to be done.

I will set up the pomp models to start 1 week before the first observation, ensuring that cases != 0.

Develop a Makefile that documents (and reproduces) the workflow for the project.

As a way to bump up the impact of the paper, look at lead times of different EWS prior to outbreaks. Recipe is:

1. Use AUC values from the fixed window analysis to determine the EWS value that serves as a threshold between non-emerging and emerging.
2. Conduct moving window EWS analysis on replicate sims.
3. Calculate length of window that the EWS is above the threshold.
4. Do 3 for many replicates and get distribution of lead times for different EWS.

Need to add section on how the SEIR simulator model is a little different than the SEIR fitting model, i.e., includes deaths etc.

In an effort to get everything in the units of yr^-1, the following things need to happen:

• Make a new time column in the observation data frame that is decimal_date(date)

• Set up the covar data frame to have a similar time column as decimal date, but do so on daily time steps to match the eventual model simulation time step

• Redo the generation of the B-spline seasonal bases to have a period of 1 yr over the daily time steps in the covar data frame

• Store observation and covar data frames separately, or nested

See recent King work for a good example.