Comparing five generative AI chatbots’ answers to LLM-generated clinical questions with medical information scientists’ evidence summaries
Journal of the Medical Library Association · 13.04.2026 · 114 (2) · 94-104
DOI: 10.5195/jmla.2026.2333 ↗Kurz gefasst
Vergleichsstudie in der Zeitschrift Journal of the Medical Library Association, online erschienen am 13.04.2026. Verfasst von Mallory N Blasingame und 6 weiteren Autor:innen. Beteiligte Einrichtung: mallory.n.blasingame@vumc.org, Center for Knowledge Management, Vanderbilt University Medical Center, Nashville, TN. Hauptthemen laut MEDLINE: Large Language Models, Information Storage and Retrieval. Die Arbeit wurde bisher 1-mal zitiert (Crossref). Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
Objective
To compare answers to clinical questions between five publicly available large language model (LLM) chatbots and information scientists.
Methods
LLMs were prompted to provide 45 PICO (patient, intervention, comparison, outcome) questions addressing treatment, prognosis, and etiology. Each question was answered by a medical information scientist and submitted to five LLM tools: ChatGPT, Gemini, Copilot, DeepSeek, and Grok-3. Key elements from the answers provided were used by pairs of information scientists to label each LLM answer as in Total Alignment, Partial Alignment, or No Alignment with the information scientist. The Partial Alignment answers were also analyzed for the inclusion of additional information.
Results
The entire LLM set of answers, 225 in total, were assessed as being in Total Alignment 20.9% of the time (n=47), in Partial Alignment 78.7% of the time (n=177), and in No Alignment 0.4% of the time (n=1). Kruskal-Wallis testing found no significant performance difference in alignment ratings between the five chatbots (p=0.46). An analysis of the partially aligned answers found a significant difference in the number of additional elements provided by the information scientists versus the chatbots per Wilcoxon-Rank Sum testing (p=0.02).
Discussion
Five chatbots did not differ significantly in their alignment with information scientists’ evidence summaries. The analysis of partially aligned answers found both chatbots and information scientists included additional information, with information scientists doing so significantly more often. An important next step will be to assess the additional information, both from the chatbots and the information scientists for validity and relevance.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- mallory.n.blasingame@vumc.org, Center for Knowledge Management, Vanderbilt University Medical Center, Nashville, TN.
- taneya.koonce@vumc.org, Center for Knowledge Management, Vanderbilt University Medical Center, Nashville, TN.
- annette.williams@vumc.org, Center for Knowledge Management, Vanderbilt University Medical Center, Nashville, TN.
- jing.su@vumc.org, Center for Knowledge Management, Vanderbilt University Medical Center, Nashville, TN.
- dario.giuse@vumc.org, Department of Biomedical Informatics, Vanderbilt University School of Medicine, Vanderbilt University Medical Center, Nashville, TN.
- poppy.krump@vumc.org, Center for Knowledge Management, Vanderbilt University Medical Center, Nashville, TN.
ORCID-Profile: Mallory N Blasingame, Taneya Y Koonce, Annette M Williams, Jing Su, Dario A Giuse, Poppy A Krump, Nunzia B Giuse
Zitieren
APA
Blasingame, M. N., Koonce, T. Y., Williams, A. M., Su, J., Giuse, D. A., Krump, P. A., & Giuse, N. B. (2026). Comparing five generative AI chatbots’ answers to LLM-generated clinical questions with medical information scientists’ evidence summaries. Journal of the Medical Library Association, 114(2), 94-104. https://doi.org/10.5195/jmla.2026.2333
Vancouver
Blasingame MN, Koonce TY, Williams AM, Su J, Giuse DA, Krump PA, et al. Comparing five generative AI chatbots’ answers to LLM-generated clinical questions with medical information scientists’ evidence summaries. Journal of the Medical Library Association. 2026;114(2):94-104. doi:10.5195/jmla.2026.2333
BibTeX
@article{Blasingame2026,
author = {Blasingame, Mallory N and Koonce, Taneya Y and Williams, Annette M and Su, Jing and Giuse, Dario A and Krump, Poppy A and Giuse, Nunzia B},
title = {Comparing five generative AI chatbots’ answers to LLM-generated clinical questions with medical information scientists’ evidence summaries},
journal = {Journal of the Medical Library Association},
year = {2026},
volume = {114},
number = {2},
pages = {94--104},
doi = {10.5195/jmla.2026.2333}
}
EUVIMED