EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Scoring epidemiological forecasts on transformed scales

Nikos I Bosse iD, Sam Abbott, Anne Cori, Edwin van Leeuwen iD, Johannes Bracher, Sebastian Funk iD

PLOS Computational Biology · 29.08.2023 · 19 (8) · e1011393

DOI: 10.1371/journal.pcbi.1011393 ↗

Kurz gefasst

Fachartikel in der Zeitschrift PLOS Computational Biology, online erschienen am 29.08.2023. Verfasst von Nikos I Bosse und 5 weiteren Autor:innen. Beteiligte Einrichtung: Department of Infectious Disease Epidemiology, London School of Hygiene & Tropical Medicine, London, United Kingdom. Hauptthemen laut MEDLINE: COVID-19, Epidemics. Die Arbeit wurde bisher 73-mal zitiert (Crossref) und nennt 38 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Wellcome Trust, Medical Research Council, National Institute for Health and Care Research (NIHR) Health Protection Research Unit.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Forecast evaluation is essential for the development of predictive epidemic models and can inform their use for public health decision-making. Common scores to evaluate epidemiological forecasts are the Continuous Ranked Probability Score (CRPS) and the Weighted Interval Score (WIS), which can be seen as measures of the absolute distance between the forecast distribution and the observation. However, applying these scores directly to predicted and observed incidence counts may not be the most appropriate due to the exponential nature of epidemic processes and the varying magnitudes of observed values across space and time. In this paper, we argue that transforming counts before applying scores such as the CRPS or WIS can effectively mitigate these difficulties and yield epidemiologically meaningful and easily interpretable results. Using the CRPS on log-transformed values as an example, we list three attractive properties: Firstly, it can be interpreted as a probabilistic version of a relative error. Secondly, it reflects how well models predicted the time-varying epidemic growth rate. And lastly, using arguments on variance-stabilizing transformations, it can be shown that under the assumption of a quadratic mean-variance relationship, the logarithmic transformation leads to expected CRPS values which are independent of the order of magnitude of the predicted quantity. Applying a transformation of log(x + 1) to data and forecasts from the European COVID-19 Forecast Hub, we find that it changes model rankings regardless of stratification by forecast date, location or target types. Situations in which models missed the beginning of upward swings are more strongly emphasised while failing to predict a downturn following a peak is less severely penalised when scoring transformed forecasts as opposed to untransformed ones. We conclude that appropriate transformations, of which the natural logarithm is only one particularly attractive option, should be considered when assessing the performance of different models in the context of infectious disease incidence.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

COVID-19EpidemicsHumansPublic HealthProbabilityRecords

Autor:innen und Institutionen

  • Department of Infectious Disease Epidemiology, London School of Hygiene & Tropical Medicine, London, United Kingdom.
  • Centre for the Mathematical Modelling of Infectious Diseases, London, United Kingdom.
  • NIHR Health Protection Research Unit in Modelling & Health Economics.
  • MRC Centre for Outbreak Analysis and Modelling, Department of Infectious Disease Epidemiology, School of Public Health, Imperial College London, London, United Kingdom.
  • Modelling & Economics Unit and NIHR Health Protection Research Unit in Modelling & Health Economics, UK Health Security Agency, London, United Kingdom.
  • Chair of Statistical Methods and Econometrics, Karlsruhe Institute of Technology, Karlsruhe, Germany.

ORCID-Profile: Nikos I Bosse, Edwin van Leeuwen, Sebastian Funk

Förderung

  • Wellcome Trust
  • Medical Research Council
  • National Institute for Health and Care Research (NIHR) Health Protection Research Unit
  • Helmholtz-Gemeinschaft
  • MRC Centre for Global Infectious Disease Analysis
  • Academy of Medical Sciences
  • Horizon 2020

Zitieren

APA

Bosse, N. I., Abbott, S., Cori, A., van Leeuwen, E., Bracher, J., & Funk, S. (2023). Scoring epidemiological forecasts on transformed scales. PLOS Computational Biology, 19(8), e1011393. https://doi.org/10.1371/journal.pcbi.1011393

Vancouver

Bosse NI, Abbott S, Cori A, van Leeuwen E, Bracher J, Funk S. Scoring epidemiological forecasts on transformed scales. PLOS Computational Biology. 2023;19(8):e1011393. doi:10.1371/journal.pcbi.1011393

BibTeX
@article{Bosse2023,
  author = {Bosse, Nikos I and Abbott, Sam and Cori, Anne and van Leeuwen, Edwin and Bracher, Johannes and Funk, Sebastian},
  title = {Scoring epidemiological forecasts on transformed scales},
  journal = {PLOS Computational Biology},
  year = {2023},
  volume = {19},
  number = {8},
  pages = {e1011393},
  doi = {10.1371/journal.pcbi.1011393}
}
Kostenlos starten →