EUVIMEDEUVIMED entdecken
ValidierungsstudieOpen Access

External Validation of a Machine Learning Model for Delirium Risk

Externe Validierung eines Machine-Learning-Modells für das Delir-Risiko

Joseph I Friedman, Prathamesh Parchure, Weijia Fu, Satyanarayana Cheertirala, Prem Timsina, Ganesh Raut, Katherine Reina, Josiane Joseph-Jimerson, Madhu Mazumdar, Robert Freeman, David L Reich, Arash Kia

JAMA Network Open · 04.11.2026 · 9 (11) · e2642142

DOI: 10.1001/jamanetworkopen.2026.42142 ↗

Auf einen Blick

In einer Qualitätsverbesserungsstudie blieb ein Fusion-Multimodal-ML-Modell mit NLP zur Delir-Risikostratifizierung bei externer Anwendung leistungsfähig. Die diagnostische Ausbeute stieg nach der Einführung, bei hoher Falsch-positiv-Rate.

Fragestellung

Wie gut funktioniert ein Fusion-Multimodal-ML-Modell mit NLP zur Delir-Risikostratifizierung in der klinischen Praxis bei externer Validierung zwischen Krankenhäusern?

Methode

Qualitätsverbesserungsstudie mit Live-Einsatz des Modells bei Patient:innen ab 60 Jahren auf Nicht-Intensivstationen; externe Validierung an einem zweiten Krankenhaus vom 1. Dezember 2024 bis 31. Dezember 2025.

Ergebnisse

AUROC 0,91 (95 %-KI 0,90-0,92) am MSM und 0,94 (95 %-KI 0,93-0,94) am MSH. Die positive Vorhersagekraft lag bei 25,9 % bzw. 19,4 %. Die monatliche Delirerkennungsrate stieg am MSM von 9,23 % auf 19,59 % (P < .001).

Einschränkungen

Hohe Falsch-positiv-Rate.

Schlussfolgerung der Autor:innen

Das ML-basierte Delir-Risikostratifizierungsmodell zeigte bei externer Anwendung ohne Retraining eine akzeptable Diskriminierung und ähnelte der Leistung am Entwicklungsstandort. Der Einsatz war mit höherer diagnostischer Ausbeute verbunden.

Kernaussagen

  • AUROC 0,91 am MSM und 0,94 am MSH
  • Positive Vorhersagekraft: 25,9 % bzw. 19,4 %
  • Delirerkennung am MSM nach Einführung von 9,23 % auf 19,59 % gestiegen

Deutsche Zusammenfassung: KI-gestützt aus dem Original-Abstract erstellt; alle Zahlen wurden automatisch mit dem Original abgeglichen. Maßgeblich ist der Original-Abstract.

Kurz gefasst

Validierungsstudie in der Zeitschrift JAMA Network Open, online erschienen am 04.11.2026. Verfasst von Joseph I Friedman und 11 weiteren Autor:innen. Beteiligte Einrichtung: Department of Psychiatry, Icahn School of Medicine at Mount Sinai, New York, New York. Hauptthemen laut MEDLINE: Delirium, Machine Learning. Die Arbeit wurde bisher 1-mal zitiert (Crossref) und nennt 51 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.

Auf einen BlickAbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Importance Although external validation of machine learning (ML)-based delirium risk models is a key stage in the model development pipeline, few published reports exist. Moreover, ML-based delirium risk models commonly degrade in performance during external validation.

Objective

To examine the performance of a fusion multimodal with natural language processing (NLP) delirium risk stratification model in live clinical practice with cross-hospital external validation. Design, Setting, and Participants This quality improvement study evaluated live clinical deployment of the model among patients 60 years or older who were admitted to non–intensive care units at Mount Sinai Hospital (MSH), with external validation performed at Mount Sinai Morningside Hospital (MSM), from December 1, 2024, to December 31, 2025. Main Outcomes and Measures Primary outcomes were performance of the risk stratification model, assessed by area under the receiver operating characteristic curve (AUROC), and monthly delirium detection rates. A 2 × 2 confusion matrix was created, and the 4 cells were used to calculate model performance metrics and compare associated diagnostic yield between premodel deployment and active model deployment periods.

Results

The overall sample included 33 095 admissions to non–intensive care units at MSM and MSH (mean [SD] age, 74.86 (9.22) years; 15 601 women [47.1%], 17 271 men [52.2%]). A total of 9020 MSM inpatient admissions were assigned a delirium risk score by the risk stratification model during live clinical operations and were compared to 21 186 MSH inpatient admissions that were assigned a risk score by the same model. Age at admission significantly differed between patients at MSM and MSH (MSM: median [IQR] age, 75.79 [68.25-83.88] years vs MSH: 72.62 [66.64-79.65] years; P &lt; .001). Patients from MSM and MSH also significantly differed by sex, race and ethnicity, surgical vs medical primary diagnosis, length of stay, and dementia prevalence rate. Among 11 909 MSM patients, patients in the pre-ML cohort (median [IQR] age, 78.00 [71.00-85.00] years) were significantly older than those in the post-ML cohort (median [IQR] age, 75.79 [68.25-83.88] years) ( P &lt; .001). The AUROC of the model was 0.91 (95% CI, 0.90-0.92) for MSM and 0.94 (95% CI, 0.93-0.94) for MSH. The positive predictive value of the model was 25.9% for MSM and 19.4% for MSH. For MSM, the mean monthly delirium detection rate significantly increased from 9.23% (95% CI, 8.07%-10.38%) to 19.59% (95% CI, 17.32%-21.85%) after model deployment ( P &lt; .001). Conclusions and Relevance In this quality improvement study, an ML-based delirium risk stratification model maintained acceptable discriminative performance during external deployment without retraining and was similar to its performance at the development site without retraining. Model deployment was also associated with increased diagnostic yield but a high false positive rate. These findings provide further evidence of the potential generalizability of the model.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

DeliriumMachine LearningHumansRisk AssessmentFemaleMaleAgedMiddle AgedPredictive Learning ModelsAged, 80 and overNatural Language ProcessingRisk Factors

Autor:innen und Institutionen

  • Department of Psychiatry, Icahn School of Medicine at Mount Sinai, New York, New York.
  • Department of Neuroscience, Icahn School of Medicine at Mount Sinai, New York, New York.
  • Institute for Healthcare Delivery Science, Icahn School of Medicine at Mount Sinai, New York, New York.
  • Nursing Administration, Mount Sinai Morningside Hospital, New York, New York.
  • Department of Nursing, The Mount Sinai Hospital, New York, New York.
  • Department of Population Health Science and Policy, Icahn School of Medicine at Mount Sinai, New York, New York.

Zitieren

APA

Friedman, J. I., Parchure, P., Fu, W., Cheertirala, S., Timsina, P., Raut, G., Reina, K., Joseph-Jimerson, J., Mazumdar, M., Freeman, R., Reich, D. L., & Kia, A. (2026). External Validation of a Machine Learning Model for Delirium Risk. JAMA Network Open, 9(11), e2642142. https://doi.org/10.1001/jamanetworkopen.2026.42142

Vancouver

Friedman JI, Parchure P, Fu W, Cheertirala S, Timsina P, Raut G, et al. External Validation of a Machine Learning Model for Delirium Risk. JAMA Network Open. 2026;9(11):e2642142. doi:10.1001/jamanetworkopen.2026.42142

BibTeX
@article{Friedman2026,
  author = {Friedman, Joseph I and Parchure, Prathamesh and Fu, Weijia and Cheertirala, Satyanarayana and Timsina, Prem and Raut, Ganesh and Reina, Katherine and Joseph-Jimerson, Josiane and Mazumdar, Madhu and Freeman, Robert and Reich, David L and Kia, Arash},
  title = {External Validation of a Machine Learning Model for Delirium Risk},
  journal = {JAMA Network Open},
  year = {2026},
  volume = {9},
  number = {11},
  pages = {e2642142},
  doi = {10.1001/jamanetworkopen.2026.42142}
}
Kostenlos starten →