Evaluating a large language model’s ability to answer clinicians’ requests for evidence summaries
Journal of the Medical Library Association · 2025
DOI: 10.5195/jmla.2025.1985 ↗Kurz gefasst
Fachartikel in der Zeitschrift Journal of the Medical Library Association, 2025. Verfasst von Mallory N Blasingame und 6 weiteren Autor:innen. Beteiligte Einrichtung: mallory.n.blasingame@vumc.org, Information Scientist & Assistant Director for Evidence Provision, Center for Knowledge Management, Vanderbilt Univers…. Hauptthemen laut MEDLINE: Artificial Intelligence, Information Storage and Retrieval, Evidence-Based Medicine. Die Arbeit wurde bisher 3-mal zitiert (Crossref). Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: NCATS NIH HHS.
Abstract (Originalsprache)
Objective
This study investigated the performance of a generative artificial intelligence (AI) tool using GPT-4 in answering clinical questions in comparison with medical librarians’ gold-standard evidence syntheses.
Methods
Questions were extracted from an in-house database of clinical evidence requests previously answered by medical librarians. Questions with multiple parts were subdivided into individual topics. A standardized prompt was developed using the COSTAR framework. Librarians submitted each question into aiChat, an internally managed chat tool using GPT-4, and recorded the responses. The summaries generated by aiChat were evaluated on whether they contained the critical elements used in the established gold-standard summary of the librarian. A subset of questions was randomly selected for verification of references provided by aiChat.
Results
Of the 216 evaluated questions, aiChat’s response was assessed as “correct” for 180 (83.3%) questions, “partially correct” for 35 (16.2%) questions, and “incorrect” for 1 (0.5%) question. No significant differences were observed in question ratings by question category (p=0.73). For a subset of 30% (n=66) of questions, 162 references were provided in the aiChat summaries, and 60 (37%) were confirmed as nonfabricated.
Conclusions
Overall, the performance of a generative AI tool was promising. However, many included references could not be independently verified, and attempts were not made to assess whether any additional concepts introduced by aiChat were factually accurate. Thus, we envision this being the first of a series of investigations designed to further our understanding of how current and future versions of generative AI can be used and integrated into medical librarians’ workflow.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- mallory.n.blasingame@vumc.org, Information Scientist & Assistant Director for Evidence Provision, Center for Knowledge Management, Vanderbilt University Medical Center, Nashville, TN.
- taneya.koonce@vumc.org, Deputy Director, Center for Knowledge Management, Vanderbilt University Medical Center, Nashville, TN.
- annette.williams@vumc.org, Senior Information Scientist and Associate Director for Metadata Management, Center for Knowledge Management, Vanderbilt University Medical Center, Nashville, TN.
- dario.giuse@vumc.org, Associate Professor, Department of Biomedical Informatics, Vanderbilt University School of Medicine and Vanderbilt University Medical Center, Nashville, TN.
- jing.su@vumc.org, Senior Information Scientist, Center for Knowledge Management, Vanderbilt University Medical Center, Nashville, TN.
- poppy.krump@vumc.org, Information Scientist, Center for Knowledge Management, Vanderbilt University Medical Center, Nashville, TN.
ORCID-Profile: Mallory N Blasingame, Taneya Y Koonce, Annette M Williams, Dario A Giuse, Jing Su, Poppy A Krump, Nunzia Bettinsoli Giuse
Förderung
- NCATS NIH HHS
Zitieren
APA
Blasingame, M. N., Koonce, T. Y., Williams, A. M., Giuse, D. A., Su, J., Krump, P. A., & Giuse, N. B. (2025). Evaluating a large language model’s ability to answer clinicians’ requests for evidence summaries. Journal of the Medical Library Association. https://doi.org/10.5195/jmla.2025.1985
Vancouver
Blasingame MN, Koonce TY, Williams AM, Giuse DA, Su J, Krump PA, et al. Evaluating a large language model’s ability to answer clinicians’ requests for evidence summaries. Journal of the Medical Library Association. 2025. doi:10.5195/jmla.2025.1985
BibTeX
@article{Blasingame2025,
author = {Blasingame, Mallory N and Koonce, Taneya Y and Williams, Annette M and Giuse, Dario A and Su, Jing and Krump, Poppy A and Giuse, Nunzia Bettinsoli},
title = {Evaluating a large language model’s ability to answer clinicians’ requests for evidence summaries},
journal = {Journal of the Medical Library Association},
year = {2025},
doi = {10.5195/jmla.2025.1985}
}
EUVIMED