Last updated: 09-02-2021 for Round 9 Scenarios.
https://covid19scenariomodelinghub.org/viz.html
Round 8: Scenario Descriptions and Model Details
Even the best models of emerging infections struggle to give accurate forecasts at time scales greater than 3-4 weeks due to unpredictable drivers such as a changing policy environment, behavior change, the development of new control measures, and stochastic events. However, policy decisions around the course of emerging infections often require projections in the time frame of months. The goal of long-term projections is to compare outbreak trajectories under different scenarios, as opposed to offering a specific, unconditional estimate of what “will” happen. As such, long-term projections can guide longer-term decision-making while short-term forecasts are more useful for situational awareness and guiding immediate response. The need for long-term epidemic projections is particularly acute in a severe pandemic, such as COVID-19, that has a large impact on the economy; for instance, economic and budget projections require estimates of outbreak trajectories in the 3-6 month time scale.
From weather to infectious diseases, it has been shown that synergizing results from multiple models gives more reliable projections than any one model alone. In the COVID-19 pandemic this approach has been exemplified by the COVID-19 Forecast Hub, which combines the results of over 30 models (see a report on the first wave of the pandemic). Further, a comparison of the impact of interventions across 17 models has illustrated how any individual model can grossly underestimate uncertainty, while ensemble projections can offer robust projections of COVID-19 the course of the epidemic under different scenarios at a 6-month time scale.
The COVID-19 Forecasting Hub provides useful and accurate short-term forecasts, but there remains a lack of publicly available model projections at 3-6 month time scale. Some single models are available online (e.g., IHME, or Imperial College), but a decade of infectious disease forecasts has demonstrated that projections from a single model are particularly risky. Single model projections are particularly problematic for emerging infections where there is much uncertainty about basic epidemiological parameters (such as the waning of immunity), the transmission process, future policies, the impact of interventions, and how the population may react to the outbreak and associated interventions. There is a need for generating long-term COVID-19 projections combining insights from different models and making them available to decision-makers, public health experts, and the general public. We plan to fill this gap by building a public COVID-19 Scenario Hub to harmonize scenario projections in the United States.
We have specified a set of scenarios and target outcomes to allow alignment of model projections for collective insights. Scenarios have been designed in consultation with academic modeling teams and government agencies (e.g., CDC).
The COVID-19 Scenario Modeling Hub is be open to any team willing to provide projections at the right temporal and spatial scales, with minimal gatekeeping. We only require that participating teams share point estimates and uncertainty bounds, along with a short model description and answers to a list of key questions about design. A major output of the projection hub would be ensemble estimates of epidemic outcomes (e.g., cases, hospitalization and/or deaths), for different time points, intervention scenarios, and US jurisdictions.
Those interested to participate should register here. Registration does not commit participants to submitting model contributions.
Model projections should be submitted via pull request to the data-processed folder of this GitHub repository. Technical instructions for submission and required file formats can be found here.
To assist with upcoming ACIP recommendations on childhood vaccination (ages 5-11), Round 9 of SMH will concentrate on evaluating the impact of childhood vaccination on COVID-19 dynamics. Results are expected to be needed by mid-September 2021.
Interpretation: These scenarios are intended to demonstrate the impact of vaccination among children ages 5 to 11. We additionally include a stress test axis which illustrates the potential impact of the emergence of a new more transmissible variant.
Model parameters defined in scenarios: With regards to the childhood vaccination axis, the data childhood vaccination begins and the state-level uptake trajectory is defined in the scenario. State-level uptake should reflect the percentage coverage increases observed in the 12 to 17-year-old age group observed since distribution to this group began on May 13, 2021. Baseline state-level age-specific vaccination data can be found here. Teams should specify in their metadata file if they use an alternative source for vaccination uptake. All assumptions about saturation over the course of the projection period should be specified in the metadata. Vaccine uptake among individuals age 12 and over should be the same in all four scenarios. Uptake in these age groups can be extrapolated from past vaccine coverage curves and vaccine hesitancy surveys (Pulse, CovidCast) with the methodology specified in the metadata. With regards to the new variant axis, the date of emergence, starting prevalence, and transmissibility increase compared to the Delta variant is specified by the scenarios.
Unconstrained model parameters: The following parameters are left to the disrection of the teams and should be noted in the metadata
- VE (infection, symptoms, severe outcomes) in all age groups
- Suggested values: Data from the REACT study suggests 60% overall VE against infection with Delta. In a study of US healthcare workers during the period of Delta variant circulation, VE was 66% against infection. Data from the UK suggests an overall VE against symptoms of 88% for Delta. VE against hospitalization ranges between 90-96% in US and UK studies against the Delta variant.
- Teams can choose different VE values for different age groups. However, chosen values should be reported in the metadata.
- Transmissibility for vaccinated and unvaccinated children, and vaccinated adults.
- Waning immunity (teams can choose to ignore waning immunity)
- Susceptibility by age
- NPIs; note that multiple jurisdictions have reinstated indoor masking and a number of schools will require masking in the fall
Outputs: In addition to the usual outputs, it would be helpful (but not required) for teams to plan to extract incident and cumulative cases, hospitalizations, and deaths for under 12 years AND 12+ years (ideal). Alternative age-specific projections will also be helpful (e.g., 0-17, 5-17). Please plan to submit quantiles for the complement of the younger age group submitted as it is not possible to extract quantiles for the older age-group by subtracting from quantiles submitted for the total population. This will allow us to provide some information on indirect effects of vaccinating children 5 to 11 years of age. Additionally, please provide population data relevant to the age groups used so appropriate rates can be calculated.
Vaccination
- Doses available:
- 50M Moderna/Pfizer 1st doses available monthly, June 2021-January 2022
- J&J no longer available (after May 2021)
- Supply has likely eclipsed demand at this stage. Number of doses are for reference and as a reminder to account for different VE by manufacturer, but no longer indicate number of doses administered. Distribution of doses by manufacturer and associated vaccine efficacy should fit within these dose bounds.
- Coverage:
- Vaccine hesitancy is expected to cause vaccination coverage to slow and eventually saturate at some level below 100%. The coverage saturation, the speed of that saturation, and heterogeneity between states (or other geospatial scales) and/or age groups are at the discretion of the modeling teams. We suggest that the teams use estimates from the Delphi group, adjusted for potential bias in respondents and the Pulse Survey overall estimates, adjusted for survey participant vaccination coverage.
- VE:
- We recommend that teams use a VE of 35% for 1st dose and 85% for second dose against symptoms for Moderna and Pfizer versus the Delta variant. These estimates reflect VE before any waning takes place.
- VE is defined here as vaccine effectiveness against symptomatic disease. Teams should make their own informed assumptions about effectiveness and impacts on other outcomes (e.g., infection, hospitalization, death).
Variant progression and transmissibility: Teams should use their own judgment to project the continued progress and transmissibility of the Delta variant across US states. Initial prevalence should be estimated or defined by the teams based on sequencing and other relevant data, preferably at the state level. Teams can set an increased severity of the Delta variant, but this should be documented in metadata.
NPI: In contrast to past scenarios, we don’t specify different levels of non-pharmaceutical interventions (NPI) use; however, teams should consider that most schools intend to return to in-person education in the fall. Teams should also note the change in CDC mask recommendations for vaccinated people in high-transmission areas on 07/27/2021.The future level of NPIs are left at the discretion of the modeling teams and should be specified in the teams’ metadata.
| Scenario | Scenario name for submission file | Scenario ID for submission file |
|---|---|---|
| Scenario A. Childhood Vaccination, No Variant | ChildVax_noVar | A-2021-09-14 |
| Scenario B. No Childhood Vaccination, No Variant | noChildVax_noVar | B-2021-09-14 |
| Scenario C. Childhood Vaccination, New Variant | ChildVax_Var | C-2021-09-14 |
| Scenario D. No Childhood Vaccination, New Variant | noChildVax_Var | D-2021-09-14 |
- Due date: September 14, 2021
- End date for fitting data: September 4 - September 11, 2021 (cut-off date at the discretion of individual teams; we’d prefer data through September 4 at least be used; no fitting should be done to data after September 11)
- Start date for scenarios: September 12, 2021 (first date of simulated transmission/outcomes)
- Simulation end date: March 12, 2022 (26-week horizon)
- Social Distancing Measures:
- Includes combined effectiveness/impact of all non-pharmaceutical interventions and behavior change.
- Current and future levels of social distancing are to be defined by the teams based on their understanding of current and planned control and behavior and expectations. Teams should consider that most jurisdictions are opening fairly quickly, and most schools intend to return to in-person education in the fall. No reactive interventions should be planned.
- Testing-Trace-Isolate: constant at baseline levels
- Masking: Included as part of “Social Distancing Measures” above.
- Immune waning and Immune escape: As defined by the scenarios.
- Vaccination:
- Pfizer / Moderna
- Vaccine efficacy (2-dose vaccines):
- B.1.1.7
- First dose: 50% against symptoms, 14 days after 1st dose
- Second dose: 90% against symptoms, 14 days after 2nd dose
- B.1.617.2
- First dose: 35% against symptoms, 14 days after 1st dose
- Second dose: 85% against symptoms, 14 days after 2nd dose
- Effectiveness and impact on infection and other outcomes (hospitalizations, deaths) is at team’s discretion and should be clearly documented in team’s metadata.
- Doses 3.5 weeks apart
- B.1.1.7
- Vaccine availability:
- December-August 13: based on data on administered doses
- August 14-February 2022: 50 million available first doses/month, with the intention of protocols being followed (100M doses/mo)
- Vaccine efficacy (2-dose vaccines):
- Johnson & Johnson
- Vaccine efficacy (1-dose):
- 70% VE against previous strains; 60% VE against B.1.1.7/B.1.617.2
- Vaccine availability:
- March-May: based on data on administered doses, with continuing at rate current on date of projection for remainder of month (~10M total administered).
- June-January: No longer available; only 10M of 20M doses administered, supply, safety, and demand issues.
- Manner for accounting for protection provided in the 10M vaccinated during March-May at team's discretion.
- Vaccine efficacy (1-dose):
- Pfizer / Moderna
- Vaccine Hesitancy: At teams' discretion.
- Delta (B.1.617.2) variant strain: At teams’ discretion. Transmission assumptions: models fit to US state-specific dynamic up until "End date for fitting data" specified above – no prescribed R0, interventions, etc.
- Transmission assumptions: models fit to US state-specific dynamic up until "End date for fitting data" specified above – no proscribed R0, interventions, etc.
- Pathogenicity assumptions: no exogenous fluctuations in pathogenicity/transmissibility beyond seasonality effects unless specified by the scenarios
- Vaccine effectiveness: see recommendations (same VE in all scenarios); assumptions regarding time required to develop immunity, age-related variation in effectiveness, duration of immunity, and additional effects of the vaccine on transmission are left to the discretion of each team
- Vaccine allocation: between-state allocation is based on population per the CDC/NAS guidelines (proportional allocation); within-state allocation and the impact of vaccine hesitancy are left to the discretion of each team
- Vaccine immunity delay: There is approximately a 14 day delay according to the Pfizer data; because we suspect the post first dose and post second dose delays may be of similar length, we do not believe there is any need to explicitly model a delay, instead groups can delay vaccine receipt by 14 days to account for it
- Vaccine uptake: See specific details.
- Vaccine roll-out: roll-out to follow ACIP recommendations unless known to be contradicted by state recommendations
- Phase 1a: health care workers, long-term care facilities
- Phase 1b: frontline essential workers, adults 75+
- Phase 1c: other essential workers, adults with high-risk conditions, adults 65-74
- NPI assumptions: NPI estimates should be based on current trends and reported planned changes.
- Database tracking of NPIs: teams may use their own data if desired, otherwise we recommend the following sources as a common starting point:
- Geographic scope: state-level and national projections
- Results: some subset of the following
- Weekly incident deaths
- Weekly cumulative deaths since start of pandemic (use JHU CSSE for baseline)
- Weekly incident reported cases
- Weekly cumulative reported cases since start of pandemic (use JHU CSSE for baseline)
- Weekly incident hospitalizations
- Weekly cumulative hospitalizations since simulation start
- Weeks will follow epi-weeks (Sun-Sat) dated by the last day of the week
- “Ground Truth”: The same data sources as the forecast hub will be used to represent “true” cases, deaths and hospitalizations. Specifically, JHU CSSE data for cases and deaths and HHS data for hospitalization.
- Metadata: We will require a brief meta-data form, TBD, from all teams.
- Uncertainty: aligned with the Forecasting Hub we ask for 0.01, 0.025, 0.05, every 5% to 0.95, 0.975, and 0.99 quantiles
- Ensemble Inclusion: at present time, in order to be included in the ensemble models need to provide a full set of quantiles
- Round 2 Scenarios
- Round 3 Scenarios
- Round 4 Scenarios
- Round 5 Scenarios
- Round 6 Scenarios
- Round 7 Scenarios
- Round 8 Scenarios
Groups interested in participating can submit model projections for each scenario in a CSV file formatted according to our specifications, and a metadata file with a description of model information. See here for technical submission requirements. Groups can submit their contributions as often as they want; the date of when a model projection was made (projection date) is recorded in the model submission file.
Model projections will have an associated model_projection_date that corresponds to the day the projection was made.
For week-ahead model projections with model_projection_date of Sunday or Monday of EW12, a 1 week ahead projection corresponds to EW12 and should have target_end_date of the Saturday of EW12. For week-ahead projections with model_projection_date of Tuesday through Saturday of EW12, a 1 week ahead projection corresponds to EW13 and should have target_end_date of the Saturday of EW13. A week-ahead projection should represent the total number of incident deaths or hospitalizations within a given epiweek (from Sunday through Saturday, inclusive) or the cumulative number of deaths reported on the Saturday of a given epiweek. We have created a csv file describing projection collection dates and dates to which projections refer to can be found. Model projection dates in the COVID-19 Scenario Modeling Hub are equivelent to the forecast dates in the COVID-19 Forecast Hub.
We will use the daily reports containing COVID-19 cases and deaths data from the JHU CSSE group as the gold standard reference data for cases and deaths in the US. We will use the distribution of the JHU data as provided by the COVIDcast Epidata API maintained by the Delphi Research Group at Carnegie Mellon University.
For COVID-19 hospitalizations, we will use the same truth data as the COVID-19 Forecast Hub, i.e., the HealthData.gov COVID-19 Reported Patient Impact and Hospital Capacity by State Timeseries. These data are released weekly although, sometimes, are updated more frequently.
A supplemental data source with daily counts that should be updated more frequently (typically daily) but does not include the full time-series is HealthData.gov COVID-19 Reported Patient Impact and Hospital Capacity by State.
Work is in progress to distribute these hospitalization data through the Covidcast Epidata API. For more information about hospitalization data, see the data section on the COVID-19 Forecast Hub.
Model projections may be submitted for any state in the US and the US at the national level.
Model projections will be represented using quantiles of predictive distributions. Similar to the COVID-19 Forecast hub, we encourage all groups to make available the following 23 quantiles for each distribution: c(0.01, 0.025, seq(0.05, 0.95, by = 0.05), 0.975, 0.99). One goal of this effort is to create probabilistic ensemble scenarios, and having high-resolution component distributions will provide data to create better ensembles.
We aim to combine model projections into an ensemble. Methods and further information will be shared when the first round of model projections have been received.
We are grateful to the teams who have generated these scenarios. The groups have made their public data available under different terms and licenses. You will find the licenses (when provided) within the model-specific folders in the data-processed directory. Please consult these licenses before using these data to ensure that you follow the terms under which these data were released.
All source code that is specific to the overall project is available under an open-source MIT license. We note that this license does NOT cover model code from the various teams or model scenario data (available under specified licenses as described above).
Those teams interested in accessing additional computational power should contact Katriona Shea at [email protected].
Teams are encouraged to share code they think will be useful to other teams via the github repo. This directory can be found in code_resources. It currently contains code to:
- Pull age-specific, state-specific, time-series data on vaccination in the US from the CDC API. get_cdc_stateagevacc.R
- Johns Hopkins ID Dynamics COVID-19 Working Group — COVID Scenario Pipeline
- Joseph C. Lemaitre (EPFL), Juan Dent Hulse (Johns Hopkins Infectious Disease Dynamics), Kyra H. Grantz (Johns Hopkins Infectious Disease Dynamics), Joshua Kaminsky (Johns Hopkins Infectious Disease Dynamics), Stephen A. Lauer (Johns Hopkins Infectious Disease Dynamics), Elizabeth C. Lee (Johns Hopkins Infectious Disease Dynamics), Justin Lessler (UNC), Hannah R. Meredith (Johns Hopkins Infectious Disease Dynamics), Javier Perez-Saez (Johns Hopkins Infectious Disease Dynamics), Shaun A. Truelove (Johns Hopkins Infectious Disease Dynamics), Claire P. Smith (Johns Hopkins Infectious Disease Dynamics), Allison Hill (Johns Hopkins Infectious Disease Dynamics), Lindsay T. Keegan (University of Utah), Kathryn Kaminsky, Sam Shah, Josh Wills, Pierre-Yves Aquilanti (Amazon Web Service), Karthik Raman (Amazon Web Services), Arun Subramaniyan (Amazon Web Services), Greg Thursam (Amazon Web Services), Anh Tran (Amazon Web Services)
- Johns Hopkins University Applied Physics Lab — Bucky
- Matt Kinsey (JHU/APL), Kate Tallaksen (JHU/APL), R.F. Obrecht (JHU/APL), Laura Asher (JHU/APL), Cash Costello (JHU/APL), Michael Kelbaugh (JHU/APL), Shelby Wilson (JHU/APL), Lauren Shin (JHU/APL), Molly Gallagher (JHU/APL), Luke Mullany (JHU/APL), Kaitlin Lovett (JHU/APL)
- Karlen Working Group — pypm
- Dean Karlen (University of Victoria and TRIUMF)
- Northeastern University MOBS Lab — GLEAM COVID
- Matteo Chinazzi (Laboratory for the Modeling of Biological and Socio-technical Systems, Northeastern University, Boston, MA), Jessica T. Davis (Laboratory for the Modeling of Biological and Socio-technical Systems, Northeastern University, Boston, MA), Kunpeng Mu (Laboratory for the Modeling of Biological and Socio-technical Systems, Northeastern University, Boston, MA), Xinyue Xiong (Laboratory for the Modeling of Biological and Socio-technical Systems, Northeastern University, Boston, MA), Ana Pastore y Piontti (Laboratory for the Modeling of Biological and Socio-technical Systems, Northeastern University, Boston, MA), Alessandro Vespignani (Laboratory for the Modeling of Biological and Socio-technical Systems, Northeastern University, Boston, MA)
- University of Southern California — SI kJalpha
- Ajitesh Srivastava (University of Southern California)
- University of Virginia — adaptive
- Przemyslaw Porebski (UVA), Srini Venkatramanan (UVA), Anniruddha Adiga (UVA), Bryan Lewis (UVA), Brian Klahn (UVA), Joseph Outten (UVA), James Schlitt (UVA), Patrick Corbett (UVA), Pyrros Alexander Telionis (UVA), Lijing Wang (UVA), Akhil Sai Peddireddy (UVA), Benjamin Hurt (UVA), Jiangzhuo Chen (UVA), Anil Vullikanti (UVA), Madhav Marathe (UVA)
- Columbia University - Age-Stratified Model
- Marta Galanti (CU), Teresa Yamana (CU), Sen Pei (CU), Jeffrey Shaman (CU)
- University of North Carolina at Charlotte - hierbin
- Shi Chen (UNC Charlotte Department of Public Health Sciences & School of Data Science), Rajib Paul (UNC Charlotte Department of Public Health Sciences and School of Data Science), Daniel Janies (UNC Charlotte Department of Bioinformatics and Genomics), Jean-Claude Thill (UNC Charlotte Department of Geography and Earth Sciences and School of Data Science)
- Institute for Health Metrics and Evaluation – IHME COVID model deaths unscaled
- Robert C Reiner, Joanne Amlag, Ryan M. Barber, James K. Collins, Peng Zheng, James Albright, Catherine M. Antony, Aleksandr Y. Aravkin, Steven D. Bachmeier, Marlena S. Bannick, Sabina Bloom, Austin Carter, Emma Castro, Kate Causey, Suman Chakrabarti, Fiona J. Charlson, Rebecca M. Cogen, Emily Combs, Xiaochen Dai, William James Dangel, Lucas Earl, Samuel B. Ewald, Maha Ezalarab, Alize J. Ferrari, Abraham Flaxman, Joseph Jon Frostad, Nancy Fullman, Emmanuela Gakidou, John Gallagher, Scott D. Glenn, Erik A. Goosmann, Jiawei He, Nathaniel J. Henry, Erin N. Hulland, Benjamin Hurst, Casey Johanns, Parkes J. Kendrick, Samantha Leigh Larson, Alice Lazzar-Atwood, Kate E. LeGrand, Haley Lescinsky, Emily Linebarger, Rafael Lozano, Rui Ma, Johan Månsson, Ana M. Mantilla Herrera, Laurie B. Marczak, Molly K. Miller-Petrie, Ali H. Mokdad, Julia Deryn Morgan, Paulami Naik, Christopher M. Odell, James K. O’Halloran, Aaron E. Osgood-Zimmerman, Samuel M. Ostroff, Maja Pasovic, Louise Penberthy, Geoffrey Phipps, David M. Pigott, Ian Pollock, Rebecca E. Ramshaw, Sofia Boston Redford, Sam Rolfe, Damian Francesco Santomauro, John R. Shackleton, David H. Shaw, Brittney S. Sheena, Aleksei Sholokhov, Reed J. D. Sorensen, Gianna Sparks, Emma Elizabeth Spurlock, Michelle L. Subart, Ruri Syailendrawati, Anna E. Torre, Christopher E. Troeger, Theo Vos, Alexandrea Watson, Stefanie Watson, Kirsten E. Wiens, Lauren Woyczynski, Liming Xu, Jize Zhang, Simon I. Hay, Stephen S. Lim & Christopher J. L. Murray
- University of Virginia - EpiHiper
- Jiangzhuo Chen (UVA), Stefan Hoops (UVA), Parantapa Bhattacharya (UVA), Dustin Machi (UVA), Bryan Lewis (UVA), Madhav Marathe (UVA)
- University of Notre Dame - FRED
- Guido Espana, Sean Cavany, Sean Moore, Alex Perkins
- Justin Lessler, University of North Carolina
- Katriona Shea, Penn State University
- Cécile Viboud, NIH Fogarty
- Shaun Truelove, Johns Hopkins University
- Rebecca Borchering, Penn State University
- Claire Smith, Johns Hopkins University
- Emily Howerton, Penn State University
- Nick Reich, University of Massachussetts at Amherst
- Wilbert Van Panhuis, University of Pittsburgh
- Harry Hochheiser, University of Pittsburgh
- Michael Runge, USGS
- Lucie Contamin, University of Pittsburgh
- John Levander, University of Pittsburgh
- Jessica Salerno, University of Pittsburgh
- J Espino, University of Pittsburgh
- Luke Mullany, Johns Hopkins University
- Kaitlin Lovett, John Hopkins University
- Michelle Qin, Harvard University
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent