epiforecasts / eval-germany-sp-nowcasting

Evaluating Semi-Parametric Nowcasts of COVID-19 Hospital Admissions in Germany

Evaluating Semi-Parametric Nowcasts of COVID-19 Hospital Admissions in Germany

This repository contains the documentation, results, and code of a project evaluating the use of a semi-parametric nowcasting approach for COVID-19 hospitalisations in Germany. See the documentation for further details. This project is part of a wider collaboration assessing a range of nowcasting methods whilst providing an ensemble nowcast of COVID-19 Hospital admissions in Germany by date of positive test. This ensemble should be used for any policy related work rather than the nowcasts provided in this repository. See here for more on this nowcasting collaboration.

Citation

If making use of the results of this analysis or reusing the analysis pipeline please cite:

If making using of the models evaluated in this analysis please also cite epinowcast:

Sam Abbott (2021). epinowcast: Hierarchical nowcasting of right censored epidemological counts, DOI: 10.5281/zenodo.5637165

A BibTeX entry for LaTeX users is also available:

@Article{, title = {epinowcast: Hierarchical nowcasting of right censored epidemological counts}, author = {Sam Abbott}, journal = {Zenodo}, year = {2021}, doi = {10.5281/zenodo.5637165}, }

Documentation

Document	Purpose
Summary	A summary of this work.
Paper	The academic paper write up of this work.
Supplementary information	The supplementary information for the write up of this work.
Real-time model evaluation	A report visualising and evaluating nowcasts from the various model configurations considered here in real-time.
Real-time method evaluation	A report visualising and evaluating nowcasts from the various methods (from this project and other groups) submitted to the Germany nowcasting hub in real-time.
Project README	Overarching project README. Includes links to resources, a summary of key files, and reproducibility information.
Analysis pipeline	The targets based analysis pipeline.
Analysis archive	An archived version of the _targets directory. Download using get_targets_archive().
Data	Documentation for input data and summarised output from the analysis.
bin	Documentation for orchestration of nowcast estimation, publishing, and archiving.
News	Dated development notes.
epinowcast	The documentation for epinowcast the R package used to implement the models evaluated here. See this case study for a simplified version of this analysis.
Germany nowcasting hub	The homepage (containing a dashboard and information) for the Germany nowcasting hub project to which nowcasts from this evaluation are submitted along with others produced by independent groups.

Key files and folders

Folder/File	Purpose
writeup	Summary paper and additional supplementary information as Rmarkdown documents.
_targets.Rmd	Analysis workflow for interactive use.
R	R functions used in the analysis and for evaluation.
data	Input data and summarised output generated by steps in the analysis.
analyses	Ad-hoc analyses not part of the overarching workflow. This includes a synthetic case study and a simplified example using Germany hospitalisation data.
.devcontainer	Resources for reproducibility using vscode and docker.

Dependencies

All dependencies can be installed using the following,

remotes::install_dev_deps()

Alternatively a docker container and image is provided. An easy way to make use of this is using the Remote development extension of vscode.

Analyses

This analysis in this repository has been implemented using the targets package and associated packages. The workflow is defined in _targets.md and can be explored interactively using _targets.Rmd Rmarkdown document. The workflow can be visualised as the following graph.

This complete analysis can be recreated using the following (note this may take quite some time even with a fairly large amount of available compute),

bash bin/update-targets.sh

Alternative the following targets functions may be used to interactively explore the workflow:

• Run the workflow sequentially.

targets::tar_make()

• Run the workflow using all available workers.

targets::tar_make_future(workers = future::availableCores())

• Explore a graph of the workflow.

targets::tar_visnetwork(targets_only = TRUE)

Watch the workflow as it runs in a shiny app.

targets::tar_watch(targets_only = TRUE)

To use our archived version of the interim results (and so avoid long run times) use the following to download it. Note that this process has not been rigorously tested across environments and so may not work seamlessly).

source(here::here("R", "targetss-archive.R"))
get_targets_archive()

eval-germany-sp-nowcasting's Issues

Error model

Currently, a negative binomial error model is used with a single overdispersion parameter. This is problematic because data with little truncation (and hence well known) is treated equally with data that is highly truncated (and hence poorly known). To some degree this uncertainty is modelled in the truncation distribution but having a single overdispersion parameter does allow additional model flexibility for which there is not much justification.

A potential option, that has been partially explored with quite poor results, would be to have an overdispersion parameter for each day of truncation. Alternatively, the simplest solution may be to assume a Poisson error model.

The effect of the single overdispersion can be seen in the posterior prediction plots where the model is quite uncertain about data with very little truncation even though these data should be very well known.

Compare to regression method

The decomposition into a hospitalisation and probability vector model used here is not the only possible nowcasting approach. A very sensible approach, though one that has slightly less nice theoretical properties is to model hospitalisations and delays jointly in a regression framework (with the outcome being new cases by time and delay). Summing over these then gives the nowcast.

Comparing these two approaches in a context where the models are otherwise the same would make it easier to understand their differences.

Similarly, the model evaluated here can also be phrased as a non-linear regression problem, for example in brms. It would be interesting to explore this implementation as it would allow a much large suite of models to be formulated though at the potential cost of some efficiency.

Non-parametric reference day hazard

Currently, the model as implemented assumes a log-normal distribution for the reporting delay prior to applying report day effects. It would be sensible to explore other the non-parametric variant of this model where report day effects are not assumed to be log normal. This would also help us understand the benefit/disadvantages of these approaches in a comparable context. This is related to epinowcast#

Daily random effects model

A daily random effects model cannot recover simulated data. Initially, I thought this was a bug but further thinking has led me to the conclusion that when the model has sufficient flexibility for the target date it will arrive at a posterior for the truncation distribution that assumes no truncation (as this gives the best fit to the observed truncated data).

This is an extreme example of this issue but potentially other flexible models in which the latest data is given to much weight will also show this issue and therefore perform poorly.

Stratify the evaluation by national/regional

Compare performance to EpiNow2::estimate_truncations

This project started as an extension of a very simple model. It may make sense to include an evaluation of this model.

Explore poor performance of complex hierarchical models

Models that are hierarchical across age groups, and particularly flexible ones, appear to perform poorly in hold out evaluation. This is somewhat surprising as models that are independent across age groups but otherwise flexible perform very well (which is not perhaps what would be expected if overfitting was the root cause). This may be attributable to a range of factors. To start exploring this potential options are:

• Check in sample performance to understand possible overfitting
• Check model formulation and model backend in epinowcast for issues.
• Visualise and explore the observations in case
• Potentially explore a hierarchical structure across locations rather than age groups to see if location differences are the key factor.

Generalise the workflow to other case studies

The workflow proposed here should be fully reproducible and generic. Given this, it would make sense to explore the models as defined here (or a related set) on another case study where nowcasting is required. An example of this is reporting of cases, deaths, and hospitalisations in the UK.