EUVIMEDUpptäck EUVIMED
BeobachtungsstudieKohortenstudieOpen access

The Impact of Evaluation Strategy on Sepsis Prediction Model Performance Metrics in Intensive Care Data: Retrospective Cohort Study

Der Einfluss der Evaluationsstrategie auf Leistungskennzahlen von Sepsis-Vorhersagemodellen in Intensivdaten: retrospektive Kohortenstudie

Dang-Khoa Do iD, Patrick Rockenschaub iD, Sebastian Daniel Boie iD, Oliver Kumpf iD, Hans-Dieter Volk iD, Felix Balzer iD, Falk von Dincklage iD, Gregor Lichtner iD

Journal of Medical Internet Research · 2026-03-24 · 28 · e72083

DOI: 10.2196/72083 ↗

Auf einen Blick

Die Leistung desselben Sepsis-Vorhersagemodells variierte je nach Evaluationsstrategie deutlich. Auf BerlinICU lag die AUROC bei 0,67 mit kontinuierlicher Evaluation und bei 0,61 mit Fixed-Horizon-Evaluation.

Fragestellung

Wie beeinflussen unterschiedliche Evaluationsstrategien die Leistungskennzahlen von Sepsis-Vorhersagemodellen in Intensivdaten?

Methode

Retrospektive, beobachtende Kohortenstudie mit Modellen aus MIMIC-IV auf BerlinICU; Vergleich von Fixed Horizon, Peak Score und Continuous Evaluation.

Ergebnisse

BerlinICU umfasste 40.132 Intensivaufnahmen über 10 Jahre, davon 4.134 septische Aufnahmen (10,3 %). AUROC: 0,67 (95 %-KI 0,66-0,68) kontinuierlich bei 6 Stunden; 0,84 (95 %-KI 0,83-0,85) auf MIMIC-IV-Testset; 0,61 (95 %-KI 0,60-0,62) Fixed Horizon.

Schlussfolgerung der Autor:innen

Die Wahl der Evaluationsstrategie hat einen erheblichen Einfluss auf die Leistungskennzahlen von Vorhersagemodellen. Kontinuierliche Evaluation bildet nach Ansicht der Autor:innen die klinische Praxis am besten ab.

Kernaussagen

  • Gleiche Modelle lieferten je nach Evaluationsstrategie unterschiedliche Kennzahlen.
  • BerlinICU: AUROC 0,67 kontinuierlich vs. 0,61 Fixed Horizon.
  • Kontinuierliche Evaluation passt laut Autor:innen am besten zur klinischen Überwachung.

Deutsche Zusammenfassung: KI-gestützt aus dem Original-Abstract erstellt; alle Zahlen wurden automatisch mit dem Original abgeglichen. Maßgeblich ist der Original-Abstract.

At a glance

Beobachtungsstudie in Journal of Medical Internet Research, published online on 2026-03-24. Written by Dang-Khoa Do and 7 co-authors. Institution: Institute of Medical Informatics, Charité - Universitätsmedizin Berlin, Corporate Member of Freie Universität Berlin and Humboldt-Universität zu Berl…. Major MEDLINE topics: Critical Care, Machine Learning, Sepsis. Cited 4 times so far (Crossref); 44 references. The full text is freely available in PubMed Central.

Auf einen BlickAbstractSubjectsBibliografiska uppgifterCiteRelated studies

Abstract (original language)

Background

The prediction of the onset of sepsis, a life-threatening condition resulting from a dysregulated response to an infection, is one of the most common prediction tasks in intensive care–related machine learning research. To assess the performance of such models, different evaluation strategies, including fixed horizon (a single prediction at a set time before onset), peak score (a single prediction using the maximum predicted risk across time), and continuous evaluation (multiple predictions assessed continuously across time), are commonly implemented, but there is no clear consensus on which approach should be used in order to provide clinically meaningful performance evaluation.

Objective

This study aimed to assess different evaluation approaches of sepsis prediction models trained on a public intensive care dataset applied to German intensive care data.

Methods

In this retrospective, observational cohort study, we assessed the efficacy of machine learning models, pretrained on the Medical Information Mart for Intensive Care IV dataset, when applied to BerlinICU, a multisite German intensive care dataset. To understand the real-world impact of implementing these models, we examined the performance variability across various evaluation strategies.

Results

The BerlinICU dataset includes 40,132 intensive care admissions spanning 10 years (2012-2021). Using the latest Sepsis-3 definition, we identified 4134 septic admissions (10.3% prevalence). Application of a temporal convolutional network model to BerlinICU yielded an area under the receiver operating characteristic curve (AUROC) of 0.67 (95% CI 0.66-0.68) for continuous evaluation with a 6-hour prediction horizon, compared with 0.84 (95% CI 0.83-0.85) on the test set of Medical Information Mart for Intensive Care IV. On BerlinICU, peak score evaluation showed a similar AUROC compared with continuous evaluation, while fixed horizon evaluation showed a reduced AUROC of 0.61 (95% CI 0.60-0.62). Onset matching had minimal impact on performance estimates using continuous evaluation or fixed horizon evaluation, but increased estimates for peak score evaluation. Performance metrics improved with shorter prediction horizons across all strategies.

Conclusions

Our results demonstrate that the choice of evaluation strategy has a significant impact on the performance metrics of intensive care prediction models. The same model applied to the same dataset yields markedly different performance metrics depending on the evaluation approach. Therefore, careful selection of the evaluation approach is essential to ensure that the interpretation of performance metrics aligns with clinical intentions and enables meaningful comparisons between studies. In our view, the continuous evaluation approach best reflects the continual monitoring of patients that is performed in real-world clinical practice. In contrast, fixed-horizon and peak score evaluation approaches may produce skewed results when not properly matching the length of stay distributions between sepsis cases and control cases. Especially for peak score evaluation, longer visits tend to produce higher maximum scores because sampling from more values increases the likelihood of capturing higher values purely by chance.

Subjects

MeSH terms (MEDLINE)

Critical CareMachine LearningSepsisHumansCohort StudiesGermanyIntensive Care UnitsPredictive Learning ModelsRetrospective Studies

Author keywords

ICUdeep learningearly predictionearly warningelectronic health recordsgeneralizabilityintensive careintensive care unitmachine learningpredictionsepsis

Authors and institutions

  • Institute of Medical Informatics, Charité - Universitätsmedizin Berlin, Corporate Member of Freie Universität Berlin and Humboldt-Universität zu Berlin, Berlin, Germany.
  • Institute of Clinical Epidemiology, Public Health, Health Economics, Medical Statistics and Informatics, Medical University of Innsbruck, Innsbruck, Austria.
  • Department of Anesthesiology and Intensive Care Medicine (CCM/CVK), Charité - Universitätsmedizin Berlin, Corporate Member of Freie Universität Berlin and Humboldt-Universität zu Berlin, Berlin, Germany.
  • Institute of Medical Immunology, Charité - Universitätsmedizin Berlin, Corporate Member of Freie Universität Berlin and Humboldt-Universität zu Berlin, Berlin, Germany.
  • Department of Anesthesia, Critical Care, Emergency and Pain Medicine, Universitätsmedizin Greifswald, Greifswald, Germany.

ORCID profiles: Dang-Khoa Do, Patrick Rockenschaub, Sebastian Daniel Boie, Oliver Kumpf, Hans-Dieter Volk, Felix Balzer, Falk von Dincklage, Gregor Lichtner

Cite

APA

Do, D. K., Rockenschaub, P., Boie, S. D., Kumpf, O., Volk, H. D., Balzer, F., von Dincklage, F., & Lichtner, G. (2026). The Impact of Evaluation Strategy on Sepsis Prediction Model Performance Metrics in Intensive Care Data: Retrospective Cohort Study. Journal of Medical Internet Research, 28, e72083. https://doi.org/10.2196/72083

Vancouver

Do DK, Rockenschaub P, Boie SD, Kumpf O, Volk HD, Balzer F, et al. The Impact of Evaluation Strategy on Sepsis Prediction Model Performance Metrics in Intensive Care Data: Retrospective Cohort Study. Journal of Medical Internet Research. 2026;28:e72083. doi:10.2196/72083

BibTeX
@article{Do2026,
  author = {Do, Dang-Khoa and Rockenschaub, Patrick and Boie, Sebastian Daniel and Kumpf, Oliver and Volk, Hans-Dieter and Balzer, Felix and von Dincklage, Falk and Lichtner, Gregor},
  title = {The Impact of Evaluation Strategy on Sepsis Prediction Model Performance Metrics in Intensive Care Data: Retrospective Cohort Study},
  journal = {Journal of Medical Internet Research},
  year = {2026},
  volume = {28},
  pages = {e72083},
  doi = {10.2196/72083}
}
Start for free →