Large language models facilitate the generation of electronic health record phenotyping algorithms
Journal of the American Medical Informatics Association · 01.09.2024 · 31 (9) · 1994-2001
DOI: 10.1093/jamia/ocae072 ↗Kurz gefasst
Fachartikel in der Zeitschrift Journal of the American Medical Informatics Association, online erschienen am 13.04.2024. Verfasst von Chao Yan und 12 weiteren Autor:innen. Beteiligte Einrichtung: Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, TN 37203, United States. Hauptthemen laut MEDLINE: Electronic Health Records, Algorithms, Phenotype. Die Arbeit wurde bisher 47-mal zitiert (Crossref) und nennt 34 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-NC-ND 4.0). Gefördert von: NIA NIH HHS, NHGRI NIH HHS, NIGMS NIH HHS.
Abstract (Originalsprache)
Objectives
Phenotyping is a core task in observational health research utilizing electronic health records (EHRs). Developing an accurate algorithm demands substantial input from domain experts, involving extensive literature review and evidence synthesis. This burdensome process limits scalability and delays knowledge discovery. We investigate the potential for leveraging large language models (LLMs) to enhance the efficiency of EHR phenotyping by generating high-quality algorithm drafts. Materials and Methods We prompted four LLMs—GPT-4 and GPT-3.5 of ChatGPT, Claude 2, and Bard—in October 2023, asking them to generate executable phenotyping algorithms in the form of SQL queries adhering to a common data model (CDM) for three phenotypes (ie, type 2 diabetes mellitus, dementia, and hypothyroidism). Three phenotyping experts evaluated the returned algorithms across several critical metrics. We further implemented the top-rated algorithms and compared them against clinician-validated phenotyping algorithms from the Electronic Medical Records and Genomics (eMERGE) network.
Results
GPT-4 and GPT-3.5 exhibited significantly higher overall expert evaluation scores in instruction following, algorithmic logic, and SQL executability, when compared to Claude 2 and Bard. Although GPT-4 and GPT-3.5 effectively identified relevant clinical concepts, they exhibited immature capability in organizing phenotyping criteria with the proper logic, leading to phenotyping algorithms that were either excessively restrictive (with low recall) or overly broad (with low positive predictive values).
Conclusion
GPT versions 3.5 and 4 are capable of drafting phenotyping algorithms by identifying relevant clinical criteria aligned with a CDM. However, expertise in informatics and clinical experience is still required to assess and further refine generated algorithms.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, TN 37203, United States.
- Department of Medicine, Vanderbilt University Medical Center, Nashville, TN 37203, United States.
- Department of Computer Science, Vanderbilt University, Nashville, TN 37203, United States.
- Department of Biostatistics, Vanderbilt University Medical Center, Nashville, TN 37203, United States.
ORCID-Profile: Chao Yan, V Eric Kerchberger
Förderung
- NIA NIH HHS
- NHGRI NIH HHS
- NIGMS NIH HHS
- NHLBI NIH HHS
- NCATS NIH HHS
- NLM NIH HHS
- National Institute of Health
Zitieren
APA
Yan, C., Ong, H. H., Grabowska, M. E., Krantz, M. S., Su, W. C., Dickson, A. L., Peterson, J. F., Feng, Q., Roden, D. M., Stein, C. M., Kerchberger, V. E., Malin, B. A., & Wei, W. Q. (2024). Large language models facilitate the generation of electronic health record phenotyping algorithms. Journal of the American Medical Informatics Association, 31(9), 1994-2001. https://doi.org/10.1093/jamia/ocae072
Vancouver
Yan C, Ong HH, Grabowska ME, Krantz MS, Su WC, Dickson AL, et al. Large language models facilitate the generation of electronic health record phenotyping algorithms. Journal of the American Medical Informatics Association. 2024;31(9):1994-2001. doi:10.1093/jamia/ocae072
BibTeX
@article{Yan2024,
author = {Yan, Chao and Ong, Henry H and Grabowska, Monika E and Krantz, Matthew S and Su, Wu-Chen and Dickson, Alyson L and Peterson, Josh F and Feng, QiPing and Roden, Dan M and Stein, C Michael and Kerchberger, V Eric and Malin, Bradley A and Wei, Wei-Qi},
title = {Large language models facilitate the generation of electronic health record phenotyping algorithms},
journal = {Journal of the American Medical Informatics Association},
year = {2024},
volume = {31},
number = {9},
pages = {1994--2001},
doi = {10.1093/jamia/ocae072}
}
EUVIMED