EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Evaluating Large Language Models (LLMs) in Information Extraction: A Case Study of Extracting Cognitive Exam Dates and Scores

Hao Zhang, Neil Jethani, Simon Jones, Nicholas Genes, Vincent J Major, Ian S Jaffe, Anthony B Cardillo, Noah Heilenbach, Nadia Fazal Ali, Luke J Bonanni, Andrew Clayburn, Zain Khera und 21 weitere

Alzheimer's & Dementia · 2024-12 · 20 (S4)

DOI: 10.1002/alz.087416 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Alzheimer's & Dementia, online erschienen am 09.01.2025. Verfasst von Hao Zhang und 32 weiteren Autor:innen. Die Arbeit wurde bisher 2-mal zitiert (Crossref). Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

Large language models (LLMs) provide powerful natural language processing capabilities in medical and clinical tasks. Evaluating LLM performance is crucial due to potential false results. In this study, we assessed ChatGPT and Llama2, two state‐of‐the‐art LLMs, in extracting information from clinical notes, focusing on cognitive tests, specifically the Mini Mental State Exam (MMSE) and Cognitive Dementia Rating (CDR).

Method

We compiled a dataset consisting of 765 clinical notes mentioning MMSE and CDR. 22 medically trained experts provided the ground truth. ChatGPT (GPT‐4, version “2023‐03‐15‐preview”) and Llama2 (“Llama‐2‐70b‐chat”) were used to extract MMSE and CDR instances with corresponding dates. Inference was successful for 742 notes. We used 20 notes for fine‐tuning and training the reviewers. The remaining 722 were assigned to reviewers, with 309 assigned to two reviewers simultaneously. Precision, sensitivity, true/false negative rates, and accuracy were calculated. For double‐reviewed notes, we qualitatively assessed the errors.

Result

The patient and note characteristics can be found in Table 1. For MMSE information extraction, ChatGPT (vs. Llama2) achieved accuracy of 83% (vs. 66.4%), sensitivity of 89.7% (vs. 69.9%), true‐negative rates of 96% (vs 60.0%), and precision of 82.7% (vs 62.2%). For CDR the results were lower overall, with accuracy of 87.1% (vs. 74.5%), sensitivity of 84.3% (vs. 39.7%), true‐negative rates of 99.8% (98.4%), and precision of 48.3% (vs. 16.1%). We qualitatively evaluated the MMSE errors of ChatGPT and Llama2 on double‐reviewed notes. Llama2 errors included 27 cases of total hallucination, 19 cases where other scores were reported instead of MMSE, 25 missed scores, and 23 cases where the wrong date was reported for the right score. In comparison, ChatGPT’s errors included only 3 cases of total hallucination, 17 cases of wrong test reported instead of MMSE, and 19 cases of reporting a wrong date.

Conclusion

ChatGPT exhibited high accuracy in extracting MMSE scores and dates, with better performance compared to Llama2. The use of LLMs could benefit dementia research and clinical care, by identifying eligible patients for treatments initialization or clinical trial enrollments. Rigorous evaluation of LLMs is crucial to understanding their capabilities and limitations.

Zitieren

APA

Zhang, H., Jethani, N., Jones, S., Genes, N., Major, V. J., Jaffe, I. S., Cardillo, A. B., Heilenbach, N., Ali, N. F., Bonanni, L. J., Clayburn, A., Khera, Z., Sadler, E. C., Prasad, J., Schlacter, J., Liu, K., Silva, B., Montgomery, S., Kim, E. J., …, & Razavian, N. (2024). Evaluating Large Language Models (LLMs) in Information Extraction: A Case Study of Extracting Cognitive Exam Dates and Scores. Alzheimer's & Dementia, 20(S4). https://doi.org/10.1002/alz.087416

Vancouver

Zhang H, Jethani N, Jones S, Genes N, Major VJ, Jaffe IS, et al. Evaluating Large Language Models (LLMs) in Information Extraction: A Case Study of Extracting Cognitive Exam Dates and Scores. Alzheimer's & Dementia. 2024;20(S4). doi:10.1002/alz.087416

BibTeX
@article{Zhang2024,
  author = {Zhang, Hao and Jethani, Neil and Jones, Simon and Genes, Nicholas and Major, Vincent J and Jaffe, Ian S and Cardillo, Anthony B and Heilenbach, Noah and Ali, Nadia Fazal and Bonanni, Luke J and Clayburn, Andrew and Khera, Zain and Sadler, Erica C and Prasad, Jaideep and Schlacter, Jamie and Liu, Kevin and Silva, Benjamin and Montgomery, Sophie and Kim, Eric J and Lester, Jacob and Hill, Theodore M and Hill, Theodore M and Avoricani, Alba and Chervonski, Ethan and Davydov, James and Small, William and Chakravartty, Eesha and Grover, Himanshu and Dodson, John and Brody, Abraham A. and Aphinyanaphongs, Yindalon and Masurkar, Arjun V. and Razavian, Narges},
  title = {Evaluating Large Language Models (LLMs) in Information Extraction: A Case Study of Extracting Cognitive Exam Dates and Scores},
  journal = {Alzheimer's & Dementia},
  year = {2024},
  volume = {20},
  number = {S4},
  doi = {10.1002/alz.087416}
}
Kostenlos starten →