EUVIMEDEUVIMED entdecken
Open Access · CC BY-NC-ND 4.0

Large language models facilitate the generation of electronic health record phenotyping algorithms

Chao Yan iD, Henry H Ong, Monika E Grabowska, Matthew S Krantz, Wu-Chen Su, Alyson L Dickson, Josh F Peterson, QiPing Feng, Dan M Roden, C Michael Stein, V Eric Kerchberger iD, Bradley A Malin und 1 weitere

Journal of the American Medical Informatics Association · 01.09.2024 · 31 (9) · 1994-2001

DOI: 10.1093/jamia/ocae072 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Journal of the American Medical Informatics Association, online erschienen am 13.04.2024. Verfasst von Chao Yan und 12 weiteren Autor:innen. Beteiligte Einrichtung: Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, TN 37203, United States. Hauptthemen laut MEDLINE: Electronic Health Records, Algorithms, Phenotype. Die Arbeit wurde bisher 47-mal zitiert (Crossref) und nennt 34 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-NC-ND 4.0). Gefördert von: NIA NIH HHS, NHGRI NIH HHS, NIGMS NIH HHS.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Objectives

Phenotyping is a core task in observational health research utilizing electronic health records (EHRs). Developing an accurate algorithm demands substantial input from domain experts, involving extensive literature review and evidence synthesis. This burdensome process limits scalability and delays knowledge discovery. We investigate the potential for leveraging large language models (LLMs) to enhance the efficiency of EHR phenotyping by generating high-quality algorithm drafts. Materials and Methods We prompted four LLMs—GPT-4 and GPT-3.5 of ChatGPT, Claude 2, and Bard—in October 2023, asking them to generate executable phenotyping algorithms in the form of SQL queries adhering to a common data model (CDM) for three phenotypes (ie, type 2 diabetes mellitus, dementia, and hypothyroidism). Three phenotyping experts evaluated the returned algorithms across several critical metrics. We further implemented the top-rated algorithms and compared them against clinician-validated phenotyping algorithms from the Electronic Medical Records and Genomics (eMERGE) network.

Results

GPT-4 and GPT-3.5 exhibited significantly higher overall expert evaluation scores in instruction following, algorithmic logic, and SQL executability, when compared to Claude 2 and Bard. Although GPT-4 and GPT-3.5 effectively identified relevant clinical concepts, they exhibited immature capability in organizing phenotyping criteria with the proper logic, leading to phenotyping algorithms that were either excessively restrictive (with low recall) or overly broad (with low positive predictive values).

Conclusion

GPT versions 3.5 and 4 are capable of drafting phenotyping algorithms by identifying relevant clinical criteria aligned with a CDM. However, expertise in informatics and clinical experience is still required to assess and further refine generated algorithms.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Electronic Health RecordsAlgorithmsPhenotypeHumansDiabetes Mellitus, Type 2DementiaHypothyroidismNatural Language Processing

Stichwörter der Autor:innen

ChatGPTelectronic health recordslarge language modelsphenotyping

Autor:innen und Institutionen

  • Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, TN 37203, United States.
  • Department of Medicine, Vanderbilt University Medical Center, Nashville, TN 37203, United States.
  • Department of Computer Science, Vanderbilt University, Nashville, TN 37203, United States.
  • Department of Biostatistics, Vanderbilt University Medical Center, Nashville, TN 37203, United States.

ORCID-Profile: Chao Yan, V Eric Kerchberger

Förderung

  • NIA NIH HHS
  • NHGRI NIH HHS
  • NIGMS NIH HHS
  • NHLBI NIH HHS
  • NCATS NIH HHS
  • NLM NIH HHS
  • National Institute of Health

Zitieren

APA

Yan, C., Ong, H. H., Grabowska, M. E., Krantz, M. S., Su, W. C., Dickson, A. L., Peterson, J. F., Feng, Q., Roden, D. M., Stein, C. M., Kerchberger, V. E., Malin, B. A., & Wei, W. Q. (2024). Large language models facilitate the generation of electronic health record phenotyping algorithms. Journal of the American Medical Informatics Association, 31(9), 1994-2001. https://doi.org/10.1093/jamia/ocae072

Vancouver

Yan C, Ong HH, Grabowska ME, Krantz MS, Su WC, Dickson AL, et al. Large language models facilitate the generation of electronic health record phenotyping algorithms. Journal of the American Medical Informatics Association. 2024;31(9):1994-2001. doi:10.1093/jamia/ocae072

BibTeX
@article{Yan2024,
  author = {Yan, Chao and Ong, Henry H and Grabowska, Monika E and Krantz, Matthew S and Su, Wu-Chen and Dickson, Alyson L and Peterson, Josh F and Feng, QiPing and Roden, Dan M and Stein, C Michael and Kerchberger, V Eric and Malin, Bradley A and Wei, Wei-Qi},
  title = {Large language models facilitate the generation of electronic health record phenotyping algorithms},
  journal = {Journal of the American Medical Informatics Association},
  year = {2024},
  volume = {31},
  number = {9},
  pages = {1994--2001},
  doi = {10.1093/jamia/ocae072}
}
Kostenlos starten →