EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Evaluating a large language model’s ability to answer clinicians’ requests for evidence summaries

Mallory N Blasingame iD, Taneya Y Koonce iD, Annette M Williams iD, Dario A Giuse iD, Jing Su iD, Poppy A Krump iD, Nunzia Bettinsoli Giuse iD

Journal of the Medical Library Association · 2025

DOI: 10.5195/jmla.2025.1985 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Journal of the Medical Library Association, 2025. Verfasst von Mallory N Blasingame und 6 weiteren Autor:innen. Beteiligte Einrichtung: mallory.n.blasingame@vumc.org, Information Scientist & Assistant Director for Evidence Provision, Center for Knowledge Management, Vanderbilt Univers…. Hauptthemen laut MEDLINE: Artificial Intelligence, Information Storage and Retrieval, Evidence-Based Medicine. Die Arbeit wurde bisher 3-mal zitiert (Crossref). Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: NCATS NIH HHS.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Objective

This study investigated the performance of a generative artificial intelligence (AI) tool using GPT-4 in answering clinical questions in comparison with medical librarians’ gold-standard evidence syntheses.

Methods

Questions were extracted from an in-house database of clinical evidence requests previously answered by medical librarians. Questions with multiple parts were subdivided into individual topics. A standardized prompt was developed using the COSTAR framework. Librarians submitted each question into aiChat, an internally managed chat tool using GPT-4, and recorded the responses. The summaries generated by aiChat were evaluated on whether they contained the critical elements used in the established gold-standard summary of the librarian. A subset of questions was randomly selected for verification of references provided by aiChat.

Results

Of the 216 evaluated questions, aiChat’s response was assessed as “correct” for 180 (83.3%) questions, “partially correct” for 35 (16.2%) questions, and “incorrect” for 1 (0.5%) question. No significant differences were observed in question ratings by question category (p=0.73). For a subset of 30% (n=66) of questions, 162 references were provided in the aiChat summaries, and 60 (37%) were confirmed as nonfabricated.

Conclusions

Overall, the performance of a generative AI tool was promising. However, many included references could not be independently verified, and attempts were not made to assess whether any additional concepts introduced by aiChat were factually accurate. Thus, we envision this being the first of a series of investigations designed to further our understanding of how current and future versions of generative AI can be used and integrated into medical librarians’ workflow.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Artificial IntelligenceInformation Storage and RetrievalEvidence-Based MedicineHumansLibrariansLibraries, MedicalLarge Language Models

Stichwörter der Autor:innen

Artificial IntelligenceBiomedical InformaticsEvidence SynthesisGenerative AIInformation ScienceLLMsLarge Language ModelsLibrary Science

Autor:innen und Institutionen

  • mallory.n.blasingame@vumc.org, Information Scientist & Assistant Director for Evidence Provision, Center for Knowledge Management, Vanderbilt University Medical Center, Nashville, TN.
  • taneya.koonce@vumc.org, Deputy Director, Center for Knowledge Management, Vanderbilt University Medical Center, Nashville, TN.
  • annette.williams@vumc.org, Senior Information Scientist and Associate Director for Metadata Management, Center for Knowledge Management, Vanderbilt University Medical Center, Nashville, TN.
  • dario.giuse@vumc.org, Associate Professor, Department of Biomedical Informatics, Vanderbilt University School of Medicine and Vanderbilt University Medical Center, Nashville, TN.
  • jing.su@vumc.org, Senior Information Scientist, Center for Knowledge Management, Vanderbilt University Medical Center, Nashville, TN.
  • poppy.krump@vumc.org, Information Scientist, Center for Knowledge Management, Vanderbilt University Medical Center, Nashville, TN.

ORCID-Profile: Mallory N Blasingame, Taneya Y Koonce, Annette M Williams, Dario A Giuse, Jing Su, Poppy A Krump, Nunzia Bettinsoli Giuse

Förderung

  • NCATS NIH HHS

Zitieren

APA

Blasingame, M. N., Koonce, T. Y., Williams, A. M., Giuse, D. A., Su, J., Krump, P. A., & Giuse, N. B. (2025). Evaluating a large language model’s ability to answer clinicians’ requests for evidence summaries. Journal of the Medical Library Association. https://doi.org/10.5195/jmla.2025.1985

Vancouver

Blasingame MN, Koonce TY, Williams AM, Giuse DA, Su J, Krump PA, et al. Evaluating a large language model’s ability to answer clinicians’ requests for evidence summaries. Journal of the Medical Library Association. 2025. doi:10.5195/jmla.2025.1985

BibTeX
@article{Blasingame2025,
  author = {Blasingame, Mallory N and Koonce, Taneya Y and Williams, Annette M and Giuse, Dario A and Su, Jing and Krump, Poppy A and Giuse, Nunzia Bettinsoli},
  title = {Evaluating a large language model’s ability to answer clinicians’ requests for evidence summaries},
  journal = {Journal of the Medical Library Association},
  year = {2025},
  doi = {10.5195/jmla.2025.1985}
}
Kostenlos starten →