External Validation of a Machine Learning Model for Delirium Risk
Externe Validierung eines Machine-Learning-Modells für das Delir-Risiko
JAMA Network Open · 04.11.2026 · 9 (11) · e2642142
DOI: 10.1001/jamanetworkopen.2026.42142 ↗Auf einen Blick
In einer Qualitätsverbesserungsstudie blieb ein Fusion-Multimodal-ML-Modell mit NLP zur Delir-Risikostratifizierung bei externer Anwendung leistungsfähig. Die diagnostische Ausbeute stieg nach der Einführung, bei hoher Falsch-positiv-Rate.
Fragestellung
Wie gut funktioniert ein Fusion-Multimodal-ML-Modell mit NLP zur Delir-Risikostratifizierung in der klinischen Praxis bei externer Validierung zwischen Krankenhäusern?
Methode
Qualitätsverbesserungsstudie mit Live-Einsatz des Modells bei Patient:innen ab 60 Jahren auf Nicht-Intensivstationen; externe Validierung an einem zweiten Krankenhaus vom 1. Dezember 2024 bis 31. Dezember 2025.
Ergebnisse
AUROC 0,91 (95 %-KI 0,90-0,92) am MSM und 0,94 (95 %-KI 0,93-0,94) am MSH. Die positive Vorhersagekraft lag bei 25,9 % bzw. 19,4 %. Die monatliche Delirerkennungsrate stieg am MSM von 9,23 % auf 19,59 % (P < .001).
Einschränkungen
Hohe Falsch-positiv-Rate.
Schlussfolgerung der Autor:innen
Das ML-basierte Delir-Risikostratifizierungsmodell zeigte bei externer Anwendung ohne Retraining eine akzeptable Diskriminierung und ähnelte der Leistung am Entwicklungsstandort. Der Einsatz war mit höherer diagnostischer Ausbeute verbunden.
Kernaussagen
- AUROC 0,91 am MSM und 0,94 am MSH
- Positive Vorhersagekraft: 25,9 % bzw. 19,4 %
- Delirerkennung am MSM nach Einführung von 9,23 % auf 19,59 % gestiegen
Deutsche Zusammenfassung: KI-gestützt aus dem Original-Abstract erstellt; alle Zahlen wurden automatisch mit dem Original abgeglichen. Maßgeblich ist der Original-Abstract.
Kurz gefasst
Validierungsstudie in der Zeitschrift JAMA Network Open, online erschienen am 04.11.2026. Verfasst von Joseph I Friedman und 11 weiteren Autor:innen. Beteiligte Einrichtung: Department of Psychiatry, Icahn School of Medicine at Mount Sinai, New York, New York. Hauptthemen laut MEDLINE: Delirium, Machine Learning. Die Arbeit wurde bisher 1-mal zitiert (Crossref) und nennt 51 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.
Abstract (Originalsprache)
Importance Although external validation of machine learning (ML)-based delirium risk models is a key stage in the model development pipeline, few published reports exist. Moreover, ML-based delirium risk models commonly degrade in performance during external validation.
Objective
To examine the performance of a fusion multimodal with natural language processing (NLP) delirium risk stratification model in live clinical practice with cross-hospital external validation. Design, Setting, and Participants This quality improvement study evaluated live clinical deployment of the model among patients 60 years or older who were admitted to non–intensive care units at Mount Sinai Hospital (MSH), with external validation performed at Mount Sinai Morningside Hospital (MSM), from December 1, 2024, to December 31, 2025. Main Outcomes and Measures Primary outcomes were performance of the risk stratification model, assessed by area under the receiver operating characteristic curve (AUROC), and monthly delirium detection rates. A 2 × 2 confusion matrix was created, and the 4 cells were used to calculate model performance metrics and compare associated diagnostic yield between premodel deployment and active model deployment periods.
Results
The overall sample included 33 095 admissions to non–intensive care units at MSM and MSH (mean [SD] age, 74.86 (9.22) years; 15 601 women [47.1%], 17 271 men [52.2%]). A total of 9020 MSM inpatient admissions were assigned a delirium risk score by the risk stratification model during live clinical operations and were compared to 21 186 MSH inpatient admissions that were assigned a risk score by the same model. Age at admission significantly differed between patients at MSM and MSH (MSM: median [IQR] age, 75.79 [68.25-83.88] years vs MSH: 72.62 [66.64-79.65] years; P < .001). Patients from MSM and MSH also significantly differed by sex, race and ethnicity, surgical vs medical primary diagnosis, length of stay, and dementia prevalence rate. Among 11 909 MSM patients, patients in the pre-ML cohort (median [IQR] age, 78.00 [71.00-85.00] years) were significantly older than those in the post-ML cohort (median [IQR] age, 75.79 [68.25-83.88] years) ( P < .001). The AUROC of the model was 0.91 (95% CI, 0.90-0.92) for MSM and 0.94 (95% CI, 0.93-0.94) for MSH. The positive predictive value of the model was 25.9% for MSM and 19.4% for MSH. For MSM, the mean monthly delirium detection rate significantly increased from 9.23% (95% CI, 8.07%-10.38%) to 19.59% (95% CI, 17.32%-21.85%) after model deployment ( P < .001). Conclusions and Relevance In this quality improvement study, an ML-based delirium risk stratification model maintained acceptable discriminative performance during external deployment without retraining and was similar to its performance at the development site without retraining. Model deployment was also associated with increased diagnostic yield but a high false positive rate. These findings provide further evidence of the potential generalizability of the model.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Autor:innen und Institutionen
- Department of Psychiatry, Icahn School of Medicine at Mount Sinai, New York, New York.
- Department of Neuroscience, Icahn School of Medicine at Mount Sinai, New York, New York.
- Institute for Healthcare Delivery Science, Icahn School of Medicine at Mount Sinai, New York, New York.
- Nursing Administration, Mount Sinai Morningside Hospital, New York, New York.
- Department of Nursing, The Mount Sinai Hospital, New York, New York.
- Department of Population Health Science and Policy, Icahn School of Medicine at Mount Sinai, New York, New York.
Zitieren
APA
Friedman, J. I., Parchure, P., Fu, W., Cheertirala, S., Timsina, P., Raut, G., Reina, K., Joseph-Jimerson, J., Mazumdar, M., Freeman, R., Reich, D. L., & Kia, A. (2026). External Validation of a Machine Learning Model for Delirium Risk. JAMA Network Open, 9(11), e2642142. https://doi.org/10.1001/jamanetworkopen.2026.42142
Vancouver
Friedman JI, Parchure P, Fu W, Cheertirala S, Timsina P, Raut G, et al. External Validation of a Machine Learning Model for Delirium Risk. JAMA Network Open. 2026;9(11):e2642142. doi:10.1001/jamanetworkopen.2026.42142
BibTeX
@article{Friedman2026,
author = {Friedman, Joseph I and Parchure, Prathamesh and Fu, Weijia and Cheertirala, Satyanarayana and Timsina, Prem and Raut, Ganesh and Reina, Katherine and Joseph-Jimerson, Josiane and Mazumdar, Madhu and Freeman, Robert and Reich, David L and Kia, Arash},
title = {External Validation of a Machine Learning Model for Delirium Risk},
journal = {JAMA Network Open},
year = {2026},
volume = {9},
number = {11},
pages = {e2642142},
doi = {10.1001/jamanetworkopen.2026.42142}
}
EUVIMED