EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Large language models are less effective at clinical prediction tasks than locally trained machine learning models

Katherine E Brown iD, Chao Yan iD, Zhuohang Li iD, Xinmeng Zhang iD, Benjamin X Collins iD, You Chen iD, Ellen Wright Clayton iD, Murat Kantarcioglu iD, Yevgeniy Vorobeychik iD, Bradley A Malin iD

Journal of the American Medical Informatics Association · 01.05.2025 · 32 (5) · 811-822

DOI: 10.1093/jamia/ocaf038 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Journal of the American Medical Informatics Association, online erschienen am 08.03.2025. Verfasst von Katherine E Brown und 9 weiteren Autor:innen. Beteiligte Einrichtung: Department of Biomedical Informatics, Vanderbilt University Medical Center (VUMC), Nashville, TN 37203, United States. Hauptthemen laut MEDLINE: Machine Learning, Electronic Health Records. Die Arbeit wurde bisher 62-mal zitiert (Crossref) und nennt 48 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: NLM NIH HHS, National Science Foundation, NIH HHS.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Objectives

To determine the extent to which current large language models (LLMs) can serve as substitutes for traditional machine learning (ML) as clinical predictors using data from electronic health records (EHRs), we investigated various factors that can impact their adoption, including overall performance, calibration, fairness, and resilience to privacy protections that reduce data fidelity. Materials and Methods We evaluated GPT-3.5, GPT-4, and traditional ML (as gradient-boosting trees) on clinical prediction tasks in EHR data from Vanderbilt University Medical Center (VUMC) and MIMIC IV. We measured predictive performance with area under the receiver operating characteristic (AUROC) and model calibration using Brier Score. To evaluate the impact of data privacy protections, we assessed AUROC when demographic variables are generalized. We evaluated algorithmic fairness using equalized odds and statistical parity across race, sex, and age of patients. We also considered the impact of using in-context learning by incorporating labeled examples within the prompt.

Results

Traditional ML [AUROC: 0.847, 0.894 (VUMC, MIMIC)] substantially outperformed GPT-3.5 (AUROC: 0.537, 0.517) and GPT-4 (AUROC: 0.629, 0.602) (with and without in-context learning) in predictive performance and output probability calibration [Brier Score (ML vs GPT-3.5 vs GPT-4): 0.134 vs 0.384 vs 0.251, 0.042 vs 0.06 vs 0.219)].

Discussion

Traditional ML is more robust than GPT-3.5 and GPT-4 in generalizing demographic information to protect privacy. GPT-4 is the fairest model according to our selected metrics but at the cost of poor model performance.

Conclusion

These findings suggest that non-fine-tuned LLMs are less effective and robust than locally trained ML for clinical prediction tasks, but they are improving across releases.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Machine LearningElectronic Health RecordsHumansFemaleROC CurveMaleAlgorithmsMiddle AgedLarge Language Models

Stichwörter der Autor:innen

clinical prediction modelsfairnesslarge language modelsprivacy

Autor:innen und Institutionen

  • Department of Biomedical Informatics, Vanderbilt University Medical Center (VUMC), Nashville, TN 37203, United States.
  • Department of Computer Science, Vanderbilt University, Nashville, TN 37212, United States.
  • Law School, Vanderbilt University, Nashville, TN 37203, United States.
  • Department of Health Policy, Vanderbilt University Medical Center (VUMC), Nashville, TN 37203, United States.
  • Department of Pediatrics, Vanderbilt University Medical Center (VUMC), Nashville, TN 37232, United States.
  • Department of Computer Science, Virginia Tech, Blacksburg, VA 24061, United States.

ORCID-Profile: Katherine E Brown, Chao Yan, Zhuohang Li, Xinmeng Zhang, Benjamin X Collins, You Chen, Ellen Wright Clayton, Murat Kantarcioglu, Yevgeniy Vorobeychik, Bradley A Malin

Förderung

  • NLM NIH HHS
  • National Science Foundation
  • NIH HHS
  • National Institutes of Health

Zitieren

APA

Brown, K. E., Yan, C., Li, Z., Zhang, X., Collins, B. X., Chen, Y., Clayton, E. W., Kantarcioglu, M., Vorobeychik, Y., & Malin, B. A. (2025). Large language models are less effective at clinical prediction tasks than locally trained machine learning models. Journal of the American Medical Informatics Association, 32(5), 811-822. https://doi.org/10.1093/jamia/ocaf038

Vancouver

Brown KE, Yan C, Li Z, Zhang X, Collins BX, Chen Y, et al. Large language models are less effective at clinical prediction tasks than locally trained machine learning models. Journal of the American Medical Informatics Association. 2025;32(5):811-822. doi:10.1093/jamia/ocaf038

BibTeX
@article{Brown2025,
  author = {Brown, Katherine E and Yan, Chao and Li, Zhuohang and Zhang, Xinmeng and Collins, Benjamin X and Chen, You and Clayton, Ellen Wright and Kantarcioglu, Murat and Vorobeychik, Yevgeniy and Malin, Bradley A},
  title = {Large language models are less effective at clinical prediction tasks than locally trained machine learning models},
  journal = {Journal of the American Medical Informatics Association},
  year = {2025},
  volume = {32},
  number = {5},
  pages = {811--822},
  doi = {10.1093/jamia/ocaf038}
}
Kostenlos starten →