The Impact of Evaluation Strategy on Sepsis Prediction Model Performance Metrics in Intensive Care Data: Retrospective Cohort Study
Der Einfluss der Evaluationsstrategie auf Leistungskennzahlen von Sepsis-Vorhersagemodellen in Intensivdaten: retrospektive Kohortenstudie
Journal of Medical Internet Research · 24.03.2026 · 28 · e72083
DOI: 10.2196/72083 ↗Auf einen Blick
Die Leistung desselben Sepsis-Vorhersagemodells variierte je nach Evaluationsstrategie deutlich. Auf BerlinICU lag die AUROC bei 0,67 mit kontinuierlicher Evaluation und bei 0,61 mit Fixed-Horizon-Evaluation.
Fragestellung
Wie beeinflussen unterschiedliche Evaluationsstrategien die Leistungskennzahlen von Sepsis-Vorhersagemodellen in Intensivdaten?
Methode
Retrospektive, beobachtende Kohortenstudie mit Modellen aus MIMIC-IV auf BerlinICU; Vergleich von Fixed Horizon, Peak Score und Continuous Evaluation.
Ergebnisse
BerlinICU umfasste 40.132 Intensivaufnahmen über 10 Jahre, davon 4.134 septische Aufnahmen (10,3 %). AUROC: 0,67 (95 %-KI 0,66-0,68) kontinuierlich bei 6 Stunden; 0,84 (95 %-KI 0,83-0,85) auf MIMIC-IV-Testset; 0,61 (95 %-KI 0,60-0,62) Fixed Horizon.
Schlussfolgerung der Autor:innen
Die Wahl der Evaluationsstrategie hat einen erheblichen Einfluss auf die Leistungskennzahlen von Vorhersagemodellen. Kontinuierliche Evaluation bildet nach Ansicht der Autor:innen die klinische Praxis am besten ab.
Kernaussagen
- Gleiche Modelle lieferten je nach Evaluationsstrategie unterschiedliche Kennzahlen.
- BerlinICU: AUROC 0,67 kontinuierlich vs. 0,61 Fixed Horizon.
- Kontinuierliche Evaluation passt laut Autor:innen am besten zur klinischen Überwachung.
Deutsche Zusammenfassung: KI-gestützt aus dem Original-Abstract erstellt; alle Zahlen wurden automatisch mit dem Original abgeglichen. Maßgeblich ist der Original-Abstract.
Kurz gefasst
Beobachtungsstudie in der Zeitschrift Journal of Medical Internet Research, online erschienen am 24.03.2026. Verfasst von Dang-Khoa Do und 7 weiteren Autor:innen. Beteiligte Einrichtung: Institute of Medical Informatics, Charité - Universitätsmedizin Berlin, Corporate Member of Freie Universität Berlin and Humboldt-Universität zu Berl…. Hauptthemen laut MEDLINE: Critical Care, Machine Learning, Sepsis. Die Arbeit wurde bisher 4-mal zitiert (Crossref) und nennt 44 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.
Themen: Intensivpflege, Sepsis
Abstract (Originalsprache)
Background
The prediction of the onset of sepsis, a life-threatening condition resulting from a dysregulated response to an infection, is one of the most common prediction tasks in intensive care–related machine learning research. To assess the performance of such models, different evaluation strategies, including fixed horizon (a single prediction at a set time before onset), peak score (a single prediction using the maximum predicted risk across time), and continuous evaluation (multiple predictions assessed continuously across time), are commonly implemented, but there is no clear consensus on which approach should be used in order to provide clinically meaningful performance evaluation.
Objective
This study aimed to assess different evaluation approaches of sepsis prediction models trained on a public intensive care dataset applied to German intensive care data.
Methods
In this retrospective, observational cohort study, we assessed the efficacy of machine learning models, pretrained on the Medical Information Mart for Intensive Care IV dataset, when applied to BerlinICU, a multisite German intensive care dataset. To understand the real-world impact of implementing these models, we examined the performance variability across various evaluation strategies.
Results
The BerlinICU dataset includes 40,132 intensive care admissions spanning 10 years (2012-2021). Using the latest Sepsis-3 definition, we identified 4134 septic admissions (10.3% prevalence). Application of a temporal convolutional network model to BerlinICU yielded an area under the receiver operating characteristic curve (AUROC) of 0.67 (95% CI 0.66-0.68) for continuous evaluation with a 6-hour prediction horizon, compared with 0.84 (95% CI 0.83-0.85) on the test set of Medical Information Mart for Intensive Care IV. On BerlinICU, peak score evaluation showed a similar AUROC compared with continuous evaluation, while fixed horizon evaluation showed a reduced AUROC of 0.61 (95% CI 0.60-0.62). Onset matching had minimal impact on performance estimates using continuous evaluation or fixed horizon evaluation, but increased estimates for peak score evaluation. Performance metrics improved with shorter prediction horizons across all strategies.
Conclusions
Our results demonstrate that the choice of evaluation strategy has a significant impact on the performance metrics of intensive care prediction models. The same model applied to the same dataset yields markedly different performance metrics depending on the evaluation approach. Therefore, careful selection of the evaluation approach is essential to ensure that the interpretation of performance metrics aligns with clinical intentions and enables meaningful comparisons between studies. In our view, the continuous evaluation approach best reflects the continual monitoring of patients that is performed in real-world clinical practice. In contrast, fixed-horizon and peak score evaluation approaches may produce skewed results when not properly matching the length of stay distributions between sepsis cases and control cases. Especially for peak score evaluation, longer visits tend to produce higher maximum scores because sampling from more values increases the likelihood of capturing higher values purely by chance.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Institute of Medical Informatics, Charité - Universitätsmedizin Berlin, Corporate Member of Freie Universität Berlin and Humboldt-Universität zu Berlin, Berlin, Germany.
- Institute of Clinical Epidemiology, Public Health, Health Economics, Medical Statistics and Informatics, Medical University of Innsbruck, Innsbruck, Austria.
- Department of Anesthesiology and Intensive Care Medicine (CCM/CVK), Charité - Universitätsmedizin Berlin, Corporate Member of Freie Universität Berlin and Humboldt-Universität zu Berlin, Berlin, Germany.
- Institute of Medical Immunology, Charité - Universitätsmedizin Berlin, Corporate Member of Freie Universität Berlin and Humboldt-Universität zu Berlin, Berlin, Germany.
- Department of Anesthesia, Critical Care, Emergency and Pain Medicine, Universitätsmedizin Greifswald, Greifswald, Germany.
ORCID-Profile: Dang-Khoa Do, Patrick Rockenschaub, Sebastian Daniel Boie, Oliver Kumpf, Hans-Dieter Volk, Felix Balzer, Falk von Dincklage, Gregor Lichtner
Zitieren
APA
Do, D. K., Rockenschaub, P., Boie, S. D., Kumpf, O., Volk, H. D., Balzer, F., von Dincklage, F., & Lichtner, G. (2026). The Impact of Evaluation Strategy on Sepsis Prediction Model Performance Metrics in Intensive Care Data: Retrospective Cohort Study. Journal of Medical Internet Research, 28, e72083. https://doi.org/10.2196/72083
Vancouver
Do DK, Rockenschaub P, Boie SD, Kumpf O, Volk HD, Balzer F, et al. The Impact of Evaluation Strategy on Sepsis Prediction Model Performance Metrics in Intensive Care Data: Retrospective Cohort Study. Journal of Medical Internet Research. 2026;28:e72083. doi:10.2196/72083
BibTeX
@article{Do2026,
author = {Do, Dang-Khoa and Rockenschaub, Patrick and Boie, Sebastian Daniel and Kumpf, Oliver and Volk, Hans-Dieter and Balzer, Felix and von Dincklage, Falk and Lichtner, Gregor},
title = {The Impact of Evaluation Strategy on Sepsis Prediction Model Performance Metrics in Intensive Care Data: Retrospective Cohort Study},
journal = {Journal of Medical Internet Research},
year = {2026},
volume = {28},
pages = {e72083},
doi = {10.2196/72083}
}
EUVIMED