Evaluating Large Language Models in extracting cognitive exam dates and scores
PLOS Digital Health · 11.12.2024 · 3 (12) · e0000685
DOI: 10.1371/journal.pdig.0000685 ↗Kurz gefasst
Fachartikel in der Zeitschrift PLOS Digital Health, online erschienen am 11.12.2024. Verfasst von Hao Zhang und 31 weiteren Autor:innen. Beteiligte Einrichtung: NYU Grossman School of Medicine, New York, New York, United States of America. Die Arbeit wurde bisher 17-mal zitiert (Crossref) und nennt 68 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: NIA NIH HHS, National Institute on Aging.
Abstract (Originalsprache)
Ensuring reliability of Large Language Models (LLMs) in clinical tasks is crucial. Our study assesses two state-of-the-art LLMs (ChatGPT and LlaMA-2) for extracting clinical information, focusing on cognitive tests like MMSE and CDR. Our data consisted of 135,307 clinical notes (Jan 12th, 2010 to May 24th, 2023) mentioning MMSE, CDR, or MoCA. After applying inclusion criteria 34,465 notes remained, of which 765 underwent ChatGPT (GPT-4) and LlaMA-2, and 22 experts reviewed the responses. ChatGPT successfully extracted MMSE and CDR instances with dates from 742 notes. We used 20 notes for fine-tuning and training the reviewers. The remaining 722 were assigned to reviewers, with 309 each assigned to two reviewers simultaneously. Inter-rater-agreement (Fleiss’ Kappa), precision, recall, true/false negative rates, and accuracy were calculated. Our study follows TRIPOD reporting guidelines for model validation. For MMSE information extraction, ChatGPT (vs. LlaMA-2) achieved accuracy of 83% (vs. 66.4%), sensitivity of 89.7% (vs. 69.9%), true-negative rates of 96% (vs 60.0%), and precision of 82.7% (vs 62.2%). For CDR the results were lower overall, with accuracy of 87.1% (vs. 74.5%), sensitivity of 84.3% (vs. 39.7%), true-negative rates of 99.8% (98.4%), and precision of 48.3% (vs. 16.1%). We qualitatively evaluated the MMSE errors of ChatGPT and LlaMA-2 on double-reviewed notes. LlaMA-2 errors included 27 cases of total hallucination, 19 cases of reporting other scores instead of MMSE, 25 missed scores, and 23 cases of reporting only the wrong date. In comparison, ChatGPT’s errors included only 3 cases of total hallucination, 17 cases of wrong test reported instead of MMSE, and 19 cases of reporting a wrong date. In this diagnostic/prognostic study of ChatGPT and LlaMA-2 for extracting cognitive exam dates and scores from clinical notes, ChatGPT exhibited high accuracy, with better performance compared to LlaMA-2. The use of LLMs could benefit dementia research and clinical care, by identifying eligible patients for treatments initialization or clinical trial enrollments. Rigorous evaluation of LLMs is crucial to understanding their capabilities and limitations.
Autor:innen und Institutionen
- NYU Grossman School of Medicine, New York, New York, United States of America.
- Rory Meyers College of Nursing, New York University, New York, New York, United States of America.
ORCID-Profile: Hao Zhang, Neil Jethani, Nicholas Genes, Vincent J Major, Ian S Jaffe, Anthony B Cardillo, Noah Heilenbach, Nadia Fazal Ali, Luke J Bonanni, Andrew J Clayburn, Zain Khera, Erica C Sadler, Jaideep Prasad, Jamie Schlacter, Kevin Liu, Benjamin Silva, Sophie Montgomery, Eric J Kim, Jacob Lester, Theodore M Hill, Alba Avoricani, Ethan Chervonski, William Small, Eesha Chakravartty, Abraham A Brody, Yindalon Aphinyanaphongs, Arjun Masurkar, Narges Razavian
Förderung
- NIA NIH HHS
- National Institute on Aging
Zitieren
APA
Zhang, H., Jethani, N., Jones, S., Genes, N., Major, V. J., Jaffe, I. S., Cardillo, A. B., Heilenbach, N., Ali, N. F., Bonanni, L. J., Clayburn, A. J., Khera, Z., Sadler, E. C., Prasad, J., Schlacter, J., Liu, K., Silva, B., Montgomery, S., Kim, E. J., …, & Razavian, N. (2024). Evaluating Large Language Models in extracting cognitive exam dates and scores. PLOS Digital Health, 3(12), e0000685. https://doi.org/10.1371/journal.pdig.0000685
Vancouver
Zhang H, Jethani N, Jones S, Genes N, Major VJ, Jaffe IS, et al. Evaluating Large Language Models in extracting cognitive exam dates and scores. PLOS Digital Health. 2024;3(12):e0000685. doi:10.1371/journal.pdig.0000685
BibTeX
@article{Zhang2024,
author = {Zhang, Hao and Jethani, Neil and Jones, Simon and Genes, Nicholas and Major, Vincent J and Jaffe, Ian S and Cardillo, Anthony B and Heilenbach, Noah and Ali, Nadia Fazal and Bonanni, Luke J and Clayburn, Andrew J and Khera, Zain and Sadler, Erica C and Prasad, Jaideep and Schlacter, Jamie and Liu, Kevin and Silva, Benjamin and Montgomery, Sophie and Kim, Eric J and Lester, Jacob and Hill, Theodore M and Avoricani, Alba and Chervonski, Ethan and Davydov, James and Small, William and Chakravartty, Eesha and Grover, Himanshu and Dodson, John A and Brody, Abraham A and Aphinyanaphongs, Yindalon and Masurkar, Arjun and Razavian, Narges},
title = {Evaluating Large Language Models in extracting cognitive exam dates and scores},
journal = {PLOS Digital Health},
year = {2024},
volume = {3},
number = {12},
pages = {e0000685},
doi = {10.1371/journal.pdig.0000685}
}
EUVIMED