Large language models are less effective at clinical prediction tasks than locally trained machine learning models
Journal of the American Medical Informatics Association · 01.05.2025 · 32 (5) · 811-822
DOI: 10.1093/jamia/ocaf038 ↗Kurz gefasst
Fachartikel in der Zeitschrift Journal of the American Medical Informatics Association, online erschienen am 08.03.2025. Verfasst von Katherine E Brown und 9 weiteren Autor:innen. Beteiligte Einrichtung: Department of Biomedical Informatics, Vanderbilt University Medical Center (VUMC), Nashville, TN 37203, United States. Hauptthemen laut MEDLINE: Machine Learning, Electronic Health Records. Die Arbeit wurde bisher 62-mal zitiert (Crossref) und nennt 48 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: NLM NIH HHS, National Science Foundation, NIH HHS.
Abstract (Originalsprache)
Objectives
To determine the extent to which current large language models (LLMs) can serve as substitutes for traditional machine learning (ML) as clinical predictors using data from electronic health records (EHRs), we investigated various factors that can impact their adoption, including overall performance, calibration, fairness, and resilience to privacy protections that reduce data fidelity. Materials and Methods We evaluated GPT-3.5, GPT-4, and traditional ML (as gradient-boosting trees) on clinical prediction tasks in EHR data from Vanderbilt University Medical Center (VUMC) and MIMIC IV. We measured predictive performance with area under the receiver operating characteristic (AUROC) and model calibration using Brier Score. To evaluate the impact of data privacy protections, we assessed AUROC when demographic variables are generalized. We evaluated algorithmic fairness using equalized odds and statistical parity across race, sex, and age of patients. We also considered the impact of using in-context learning by incorporating labeled examples within the prompt.
Results
Traditional ML [AUROC: 0.847, 0.894 (VUMC, MIMIC)] substantially outperformed GPT-3.5 (AUROC: 0.537, 0.517) and GPT-4 (AUROC: 0.629, 0.602) (with and without in-context learning) in predictive performance and output probability calibration [Brier Score (ML vs GPT-3.5 vs GPT-4): 0.134 vs 0.384 vs 0.251, 0.042 vs 0.06 vs 0.219)].
Discussion
Traditional ML is more robust than GPT-3.5 and GPT-4 in generalizing demographic information to protect privacy. GPT-4 is the fairest model according to our selected metrics but at the cost of poor model performance.
Conclusion
These findings suggest that non-fine-tuned LLMs are less effective and robust than locally trained ML for clinical prediction tasks, but they are improving across releases.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Department of Biomedical Informatics, Vanderbilt University Medical Center (VUMC), Nashville, TN 37203, United States.
- Department of Computer Science, Vanderbilt University, Nashville, TN 37212, United States.
- Law School, Vanderbilt University, Nashville, TN 37203, United States.
- Department of Health Policy, Vanderbilt University Medical Center (VUMC), Nashville, TN 37203, United States.
- Department of Pediatrics, Vanderbilt University Medical Center (VUMC), Nashville, TN 37232, United States.
- Department of Computer Science, Virginia Tech, Blacksburg, VA 24061, United States.
ORCID-Profile: Katherine E Brown, Chao Yan, Zhuohang Li, Xinmeng Zhang, Benjamin X Collins, You Chen, Ellen Wright Clayton, Murat Kantarcioglu, Yevgeniy Vorobeychik, Bradley A Malin
Förderung
- NLM NIH HHS
- National Science Foundation
- NIH HHS
- National Institutes of Health
Zitieren
APA
Brown, K. E., Yan, C., Li, Z., Zhang, X., Collins, B. X., Chen, Y., Clayton, E. W., Kantarcioglu, M., Vorobeychik, Y., & Malin, B. A. (2025). Large language models are less effective at clinical prediction tasks than locally trained machine learning models. Journal of the American Medical Informatics Association, 32(5), 811-822. https://doi.org/10.1093/jamia/ocaf038
Vancouver
Brown KE, Yan C, Li Z, Zhang X, Collins BX, Chen Y, et al. Large language models are less effective at clinical prediction tasks than locally trained machine learning models. Journal of the American Medical Informatics Association. 2025;32(5):811-822. doi:10.1093/jamia/ocaf038
BibTeX
@article{Brown2025,
author = {Brown, Katherine E and Yan, Chao and Li, Zhuohang and Zhang, Xinmeng and Collins, Benjamin X and Chen, You and Clayton, Ellen Wright and Kantarcioglu, Murat and Vorobeychik, Yevgeniy and Malin, Bradley A},
title = {Large language models are less effective at clinical prediction tasks than locally trained machine learning models},
journal = {Journal of the American Medical Informatics Association},
year = {2025},
volume = {32},
number = {5},
pages = {811--822},
doi = {10.1093/jamia/ocaf038}
}
EUVIMED