EUVIMEDEUVIMED entdecken
VergleichsstudieBeobachtungsstudieOpen Access

Evaluating the reference accuracy of large language models in radiology: a comparative study across subspecialties

Yasin Celal Güneş iD, Turay Cesur iD, Eren Çamur iD

Diagnostic and Interventional Radiology · 12.05.2025

DOI: 10.4274/dir.2025.253101 ↗

Kurz gefasst

Vergleichsstudie in der Zeitschrift Diagnostic and Interventional Radiology, online erschienen am 12.05.2025. Verfasst von Yasin Celal Güneş und 2 weiteren Autor:innen. Beteiligte Einrichtung: Kırıkkale Yüksek İhtisas Hospital, Clinic of Radiology, Kırıkkale, Türkiye. Hauptthemen laut MEDLINE: Radiology, Language. Die Arbeit wurde bisher 4-mal zitiert (Crossref) und nennt 1 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

PURPOSE: This study aimed to compare six large language models (LLMs) [Chat Generative Pre-trained Transformer (ChatGPT)o1-preview, ChatGPT-4o, ChatGPT-4o with canvas, Google Gemini 1.5 Pro, Claude 3.5 Sonnet, and Claude 3 Opus] in generating radiology references, assessing accuracy, fabrication, and bibliographic completeness. METHODS: In this cross-sectional observational study, 120 open-ended questions were administered across eight radiology subspecialties (neuroradiology, abdominal, musculoskeletal, thoracic, pediatric, cardiac, head and neck, and interventional radiology), with 15 questions per subspecialty. Each question prompted the LLMs to provide responses containing four references with in-text citations and complete bibliographic details (authors, title, journal, publication year/month, volume, issue, page numbers, and PubMed Identifier). References were verified using Medline, Google Scholar, the Directory of Open Access Journals, and web searches. Each bibliographic element was scored for correctness, and a composite final score [(FS): 0-36] was calculated by summing the correct elements and multiplying this by a 5-point verification score for content relevance. The FS values were then categorized into a 5-point Likert scale reference accuracy score (RAS: 0 = fabricated; 4 = fully accurate). Non-parametric tests (Kruskal-Wallis, Tamhane's T2, Wilcoxon signed-rank test with Bonferroni correction) were used for statistical comparisons. RESULTS: Claude 3.5 Sonnet demonstrated the highest reference accuracy, with 80.8% fully accurate references (RAS 4) and a fabrication rate of 3.1%, significantly outperforming all other models (P < 0.001). Claude 3 Opus ranked second, achieving 59.6% fully accurate references and a fabrication rate of 18.3% (P < 0.001). ChatGPT-based models (ChatGPT-4o, ChatGPT-4o with canvas, and ChatGPT o1-preview) exhibited moderate accuracy, with fabrication rates ranging from 27.7% to 52.9% and

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

RadiologyLanguageHumansCross-Sectional StudiesLarge Language Models

Stichwörter der Autor:innen

ChatGPT o1-previewClaude 3.5 SonnetReferencecitationlarge language models

Autor:innen und Institutionen

  • Kırıkkale Yüksek İhtisas Hospital, Clinic of Radiology, Kırıkkale, Türkiye.
  • Mamak State Hospital, Clinic of Radiology, Ankara, Türkiye.
  • Ankara 29 May State Hospital, Clinic of Radiology, Ankara, Türkiye.

ORCID-Profile: Yasin Celal Güneş, Turay Cesur, Eren Çamur

Zitieren

APA

Güneş, Y. C., Cesur, T., & Çamur, E. (2025). Evaluating the reference accuracy of large language models in radiology: a comparative study across subspecialties. Diagnostic and Interventional Radiology. https://doi.org/10.4274/dir.2025.253101

Vancouver

Güneş YC, Cesur T, Çamur E. Evaluating the reference accuracy of large language models in radiology: a comparative study across subspecialties. Diagnostic and Interventional Radiology. 2025. doi:10.4274/dir.2025.253101

BibTeX
@article{Gne2025,
  author = {Güneş, Yasin Celal and Cesur, Turay and Çamur, Eren},
  title = {Evaluating the reference accuracy of large language models in radiology: a comparative study across subspecialties},
  journal = {Diagnostic and Interventional Radiology},
  year = {2025},
  doi = {10.4274/dir.2025.253101}
}
Kostenlos starten →