EUVIMEDEUVIMED entdecken
VergleichsstudieOpen Access

A Comparative Evaluation of Large Language Model Utility in Neuroimaging Clinical Decision Support

Luke Miller, Peter Kamel, Jigar Patel, Jay Agrawal, Min Zhan, Nathan Bumbarger, Kenneth Wang

Journal of Imaging Informatics in Medicine · 2024

DOI: 10.1007/s10278-024-01161-3 ↗

Kurz gefasst

Vergleichsstudie in der Zeitschrift Journal of Imaging Informatics in Medicine, 2024. Verfasst von Luke Miller und 6 weiteren Autor:innen. Beteiligte Einrichtung: Department of Radiology, University of Maryland Medical Center, Baltimore, MD, USA. Luke.Miller@umm.edu. Hauptthemen laut MEDLINE: Neuroimaging, Decision Support Systems, Clinical, Natural Language Processing. Die Arbeit wurde bisher 3-mal zitiert (Crossref) und nennt 29 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Imaging utilization has increased dramatically in recent years, and at least some of these studies are not appropriate for the clinical scenario. The development of large language models (LLMs) may address this issue by providing a more accessible reference resource for ordering providers, but their relative performance is currently understudied. Evaluate and compare the relative appropriateness and usefulness of imaging recommendations generated by eight publicly available models in response to neuroradiology clinical scenarios. Twenty-four common neuroradiology clinical scenarios were selected which often yield suboptimal imaging utilization. Questions were crafted to assess the ability of LLMs to provide accurate and actionable advice. The LLMs were assessed in August 2023 using natural-language 1-2 sentence queries requesting advice about optimal image ordering given certain clinical parameters. Eight of the most well-known LLMs were chosen for evaluation: ChatGPT, GPT4, Bard (Versions 1 and 2), Bing Chat, Llama 2, Perplexity, and Claude. The models were graded by three fellowship-trained neuroradiologists on whether their advice was "optimal" or "not optimal" according to the ACR Appropriateness Criteria or the New Orleans Head CT Criteria. The raters also ranked the models based on the appropriateness, helpfulness, concision, and source-citations in their response. The models varied in their ability to deliver an "optimal" recommendation based on these scenarios as follows: ChatGPT (20/24), GPT4 (23/24), Bard 1 (13/24), Bard 2 (14/24), Bing Chat (14/24), Llama (5/24), Perplexity (19/24), and Claude (19/24). The median ranks of the LLMs were as follows: ChatGPT (3), GPT4 (1.5), Bard 1 (4.5), Bard 2 (5), Bing Chat (6), Llama (7.5), Perplexity (4), and Claude (3). Characteristic errors are described and discussed. GPT-4, ChatGPT, and Claude generally outperformed Bard, Bing Chat, and Llama 2. This study evaluates the performance of a greater variety of publicly available LLMs in settings that more closely mimic real-world use cases as well as discussing the practical challenges of doing so. This is the first study to evaluate and compare a wide range of publicly available LLMs to determine appropriateness of their neuroradiology imaging recommendations.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

NeuroimagingDecision Support Systems, ClinicalNatural Language ProcessingHumansLanguageLarge Language Models

Stichwörter der Autor:innen

BardChatGPTGPT4Imaging utilizationLLM

Autor:innen und Institutionen

  • Department of Radiology, University of Maryland Medical Center, Baltimore, MD, USA. Luke.Miller@umm.edu.
  • Department of Radiology, University of Maryland Medical Center, Baltimore, MD, USA.
  • Department of Radiology, Baltimore VA Medical Center, Baltimore, MD, USA.
  • Epidemiology and Public Health, University of Maryland School of Medicine, Baltimore, USA.

Zitieren

APA

Miller, L., Kamel, P., Patel, J., Agrawal, J., Zhan, M., Bumbarger, N., & Wang, K. (2024). A Comparative Evaluation of Large Language Model Utility in Neuroimaging Clinical Decision Support. Journal of Imaging Informatics in Medicine. https://doi.org/10.1007/s10278-024-01161-3

Vancouver

Miller L, Kamel P, Patel J, Agrawal J, Zhan M, Bumbarger N, et al. A Comparative Evaluation of Large Language Model Utility in Neuroimaging Clinical Decision Support. Journal of Imaging Informatics in Medicine. 2024. doi:10.1007/s10278-024-01161-3

BibTeX
@article{Miller2024,
  author = {Miller, Luke and Kamel, Peter and Patel, Jigar and Agrawal, Jay and Zhan, Min and Bumbarger, Nathan and Wang, Kenneth},
  title = {A Comparative Evaluation of Large Language Model Utility in Neuroimaging Clinical Decision Support},
  journal = {Journal of Imaging Informatics in Medicine},
  year = {2024},
  doi = {10.1007/s10278-024-01161-3}
}
Kostenlos starten →