EUVIMEDEUVIMED entdecken
VergleichsstudieOpen Access · CC BY 4.0

Comparing five generative AI chatbots’ answers to LLM-generated clinical questions with medical information scientists’ evidence summaries

Mallory N Blasingame iD, Taneya Y Koonce iD, Annette M Williams iD, Jing Su iD, Dario A Giuse iD, Poppy A Krump iD, Nunzia B Giuse iD

Journal of the Medical Library Association · 13.04.2026 · 114 (2) · 94-104

DOI: 10.5195/jmla.2026.2333 ↗

Kurz gefasst

Vergleichsstudie in der Zeitschrift Journal of the Medical Library Association, online erschienen am 13.04.2026. Verfasst von Mallory N Blasingame und 6 weiteren Autor:innen. Beteiligte Einrichtung: mallory.n.blasingame@vumc.org, Center for Knowledge Management, Vanderbilt University Medical Center, Nashville, TN. Hauptthemen laut MEDLINE: Large Language Models, Information Storage and Retrieval. Die Arbeit wurde bisher 1-mal zitiert (Crossref). Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Objective

To compare answers to clinical questions between five publicly available large language model (LLM) chatbots and information scientists.

Methods

LLMs were prompted to provide 45 PICO (patient, intervention, comparison, outcome) questions addressing treatment, prognosis, and etiology. Each question was answered by a medical information scientist and submitted to five LLM tools: ChatGPT, Gemini, Copilot, DeepSeek, and Grok-3. Key elements from the answers provided were used by pairs of information scientists to label each LLM answer as in Total Alignment, Partial Alignment, or No Alignment with the information scientist. The Partial Alignment answers were also analyzed for the inclusion of additional information.

Results

The entire LLM set of answers, 225 in total, were assessed as being in Total Alignment 20.9% of the time (n=47), in Partial Alignment 78.7% of the time (n=177), and in No Alignment 0.4% of the time (n=1). Kruskal-Wallis testing found no significant performance difference in alignment ratings between the five chatbots (p=0.46). An analysis of the partially aligned answers found a significant difference in the number of additional elements provided by the information scientists versus the chatbots per Wilcoxon-Rank Sum testing (p=0.02).

Discussion

Five chatbots did not differ significantly in their alignment with information scientists’ evidence summaries. The analysis of partially aligned answers found both chatbots and information scientists included additional information, with information scientists doing so significantly more often. An important next step will be to assess the additional information, both from the chatbots and the information scientists for validity and relevance.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Large Language ModelsInformation Storage and RetrievalGenerative Artificial IntelligenceHumans

Stichwörter der Autor:innen

LLMsLarge Language Modelsartificial intelligencebiomedical informaticschatbotsevidence synthesisgenerative AIinformation sciencelibrary science

Autor:innen und Institutionen

  • mallory.n.blasingame@vumc.org, Center for Knowledge Management, Vanderbilt University Medical Center, Nashville, TN.
  • taneya.koonce@vumc.org, Center for Knowledge Management, Vanderbilt University Medical Center, Nashville, TN.
  • annette.williams@vumc.org, Center for Knowledge Management, Vanderbilt University Medical Center, Nashville, TN.
  • jing.su@vumc.org, Center for Knowledge Management, Vanderbilt University Medical Center, Nashville, TN.
  • dario.giuse@vumc.org, Department of Biomedical Informatics, Vanderbilt University School of Medicine, Vanderbilt University Medical Center, Nashville, TN.
  • poppy.krump@vumc.org, Center for Knowledge Management, Vanderbilt University Medical Center, Nashville, TN.

ORCID-Profile: Mallory N Blasingame, Taneya Y Koonce, Annette M Williams, Jing Su, Dario A Giuse, Poppy A Krump, Nunzia B Giuse

Zitieren

APA

Blasingame, M. N., Koonce, T. Y., Williams, A. M., Su, J., Giuse, D. A., Krump, P. A., & Giuse, N. B. (2026). Comparing five generative AI chatbots’ answers to LLM-generated clinical questions with medical information scientists’ evidence summaries. Journal of the Medical Library Association, 114(2), 94-104. https://doi.org/10.5195/jmla.2026.2333

Vancouver

Blasingame MN, Koonce TY, Williams AM, Su J, Giuse DA, Krump PA, et al. Comparing five generative AI chatbots’ answers to LLM-generated clinical questions with medical information scientists’ evidence summaries. Journal of the Medical Library Association. 2026;114(2):94-104. doi:10.5195/jmla.2026.2333

BibTeX
@article{Blasingame2026,
  author = {Blasingame, Mallory N and Koonce, Taneya Y and Williams, Annette M and Su, Jing and Giuse, Dario A and Krump, Poppy A and Giuse, Nunzia B},
  title = {Comparing five generative AI chatbots’ answers to LLM-generated clinical questions with medical information scientists’ evidence summaries},
  journal = {Journal of the Medical Library Association},
  year = {2026},
  volume = {114},
  number = {2},
  pages = {94--104},
  doi = {10.5195/jmla.2026.2333}
}
Kostenlos starten →