EUVIMEDEUVIMED entdecken
Open Access · CC BY-NC 3.0

Diagnostic performance of multimodal large language models in radiological quiz cases: the effects of prompt engineering and input conditions

Taewon Han, Woo Kyoung Jeong, Jaeseung Shin

Ultrasonography · 01.05.2025 · 44 (3) · 220-231

DOI: 10.14366/usg.25012 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Ultrasonography, erschienen am 01.05.2025. Verfasst von Taewon Han und 2 weiteren Autor:innen. Beteiligte Einrichtung: Department of Radiology and Center for Imaging Sciences, Samsung Medical Center, Sungkyunkwan University School of Medicine, Seoul, Korea. Die Arbeit wurde bisher 19-mal zitiert (Crossref) und nennt 37 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-NC 3.0).

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Purpose

This study aimed to evaluate the diagnostic accuracy of three multimodal large language models (LLMs) in radiological image interpretation and to assess the impact of prompt engineering strategies and input conditions.

Methods

This study analyzed 67 radiological quiz cases from the Korean Society of Ultrasound in Medicine. Three multimodal LLMs (Claude 3.5 Sonnet, GPT-4o, and Gemini-1.5-Pro-002) were evaluated using six types of prompts (basic [without system prompt], original [specific instructions], chain-of-thought, reflection, multiagent, and artificial intelligence [AI]–generated). Performance was assessed across various factors, including tumor versus non-tumor status, case rarity, difficulty, and knowledge cutoff dates. A subgroup analysis compared diagnostic accuracy between imaging-only inputs and combined imaging-descriptive text inputs.

Results

With imaging-only inputs, Claude 3.5 Sonnet achieved the highest overall accuracy (46.3%, 186/402), followed by GPT-4o (43.5%, 175/402) and Gemini-1.5-Pro-002 (39.8%, 160/402). AI-generated prompts yielded superior combined accuracy across all three models, with improvements over the basic (5.5%, P=0.035), chain-of-thought (4.0%, P=0.169), and multiagent prompts (3.5%, P=0.248). The integration of descriptive text significantly enhanced diagnostic accuracy for Claude 3.5 Sonnet (46.3% to 66.2%, P

Schlagwörter

Stichwörter der Autor:innen

Artificial intelligenceDiagnostic imagingMedical informaticsNatural language processing

Autor:innen und Institutionen

  • Department of Radiology and Center for Imaging Sciences, Samsung Medical Center, Sungkyunkwan University School of Medicine, Seoul, Korea.

Zitieren

APA

Han, T., Jeong, W. K., & Shin, J. (2025). Diagnostic performance of multimodal large language models in radiological quiz cases: the effects of prompt engineering and input conditions. Ultrasonography, 44(3), 220-231. https://doi.org/10.14366/usg.25012

Vancouver

Han T, Jeong WK, Shin J. Diagnostic performance of multimodal large language models in radiological quiz cases: the effects of prompt engineering and input conditions. Ultrasonography. 2025;44(3):220-231. doi:10.14366/usg.25012

BibTeX
@article{Han2025,
  author = {Han, Taewon and Jeong, Woo Kyoung and Shin, Jaeseung},
  title = {Diagnostic performance of multimodal large language models in radiological quiz cases: the effects of prompt engineering and input conditions},
  journal = {Ultrasonography},
  year = {2025},
  volume = {44},
  number = {3},
  pages = {220--231},
  doi = {10.14366/usg.25012}
}
Kostenlos starten →