Diagnostic performance of multimodal large language models in radiological quiz cases: the effects of prompt engineering and input conditions
Ultrasonography · 01.05.2025 · 44 (3) · 220-231
DOI: 10.14366/usg.25012 ↗Kurz gefasst
Fachartikel in der Zeitschrift Ultrasonography, erschienen am 01.05.2025. Verfasst von Taewon Han und 2 weiteren Autor:innen. Beteiligte Einrichtung: Department of Radiology and Center for Imaging Sciences, Samsung Medical Center, Sungkyunkwan University School of Medicine, Seoul, Korea. Die Arbeit wurde bisher 19-mal zitiert (Crossref) und nennt 37 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-NC 3.0).
Abstract (Originalsprache)
Purpose
This study aimed to evaluate the diagnostic accuracy of three multimodal large language models (LLMs) in radiological image interpretation and to assess the impact of prompt engineering strategies and input conditions.
Methods
This study analyzed 67 radiological quiz cases from the Korean Society of Ultrasound in Medicine. Three multimodal LLMs (Claude 3.5 Sonnet, GPT-4o, and Gemini-1.5-Pro-002) were evaluated using six types of prompts (basic [without system prompt], original [specific instructions], chain-of-thought, reflection, multiagent, and artificial intelligence [AI]–generated). Performance was assessed across various factors, including tumor versus non-tumor status, case rarity, difficulty, and knowledge cutoff dates. A subgroup analysis compared diagnostic accuracy between imaging-only inputs and combined imaging-descriptive text inputs.
Results
With imaging-only inputs, Claude 3.5 Sonnet achieved the highest overall accuracy (46.3%, 186/402), followed by GPT-4o (43.5%, 175/402) and Gemini-1.5-Pro-002 (39.8%, 160/402). AI-generated prompts yielded superior combined accuracy across all three models, with improvements over the basic (5.5%, P=0.035), chain-of-thought (4.0%, P=0.169), and multiagent prompts (3.5%, P=0.248). The integration of descriptive text significantly enhanced diagnostic accuracy for Claude 3.5 Sonnet (46.3% to 66.2%, P
Schlagwörter
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Department of Radiology and Center for Imaging Sciences, Samsung Medical Center, Sungkyunkwan University School of Medicine, Seoul, Korea.
Zitieren
APA
Han, T., Jeong, W. K., & Shin, J. (2025). Diagnostic performance of multimodal large language models in radiological quiz cases: the effects of prompt engineering and input conditions. Ultrasonography, 44(3), 220-231. https://doi.org/10.14366/usg.25012
Vancouver
Han T, Jeong WK, Shin J. Diagnostic performance of multimodal large language models in radiological quiz cases: the effects of prompt engineering and input conditions. Ultrasonography. 2025;44(3):220-231. doi:10.14366/usg.25012
BibTeX
@article{Han2025,
author = {Han, Taewon and Jeong, Woo Kyoung and Shin, Jaeseung},
title = {Diagnostic performance of multimodal large language models in radiological quiz cases: the effects of prompt engineering and input conditions},
journal = {Ultrasonography},
year = {2025},
volume = {44},
number = {3},
pages = {220--231},
doi = {10.14366/usg.25012}
}
EUVIMED