Comparative evaluation of large language models on multiple-choice and image-based rheumatology questions
Rheumatology International · 08.12.2025 · 46 (1)
DOI: 10.1007/s00296-025-06053-5 ↗Kurz gefasst
Vergleichsstudie in der Zeitschrift Rheumatology International, online erschienen am 08.12.2025. Verfasst von Pannathorn Nakaphan und 4 weiteren Autor:innen. Beteiligte Einrichtung: Department of Internal Medicine, University of Missouri-Kansas City, Kansas City, MO, USA. pannathorn0769@gmail.com. Hauptthemen laut MEDLINE: Large Language Models, Rheumatology. Die Arbeit wurde bisher 9-mal zitiert (Crossref) und nennt 34 Literaturangaben.
Abstract (Originalsprache)
Large language models (LLMs) are increasingly used in medical education and clinical decision support, including applications in rheumatology. We evaluated seven publicly accessible LLM tools, ChatGPT (GPT-3.5 and GPT-4.0), Claude Sonnet 4, Gemini, Perplexity AI, DeepSeek, and OpenEvidence, using 50 multiple-choice questions (MCQs) and 25 image-based diagnostic prompts. We assessed accuracy, self-reported confidence, and hallucination rates. In MCQs, DeepSeek achieved the highest accuracy (96%), followed by Claude (94%), GPT-3.5 (92%), GPT-4.0 (92%), Gemini (92%), OpenEvidence (92%), and Perplexity (90%). Image-based performance was lower and more variable, ranging from 16% (Claude) to 56% (Gemini). All models showed significantly reduced odds of correct responses to image questions compared to MCQs (p < 0.01). Claude performed significantly worse than GPT-3.5 on image-based questions (OR 0.24; 95% CI: 0.06–0.86, p = 0.04); no model significantly outperformed GPT-3.5 on MCQs. Confidence scores remained high across all models, ranging from 7 to 10 for MCQs and 8 to 10 for image-based questions. Hallucinations were rare for MCQs (n = 3; 1 Gemini, 2 Perplexity) but common in image responses, ranging from 40% (Gemini) to 84% (Claude). Publicly available LLMs demonstrate high accuracy on text-based rheumatology questions but show limited capability in image interpretation. High confidence in incorrect image responses and frequent hallucinations highlight the need for caution when integrating these tools into clinical education or decision-making.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Department of Internal Medicine, University of Missouri-Kansas City, Kansas City, MO, USA. pannathorn0769@gmail.com.
- Department of Medicine, Richmond University Medical Center, New York, USA.
- Department of Internal Medicine, University of Connecticut, Farmington, CT, USA.
- Department of Medicine, Texas Tech University Health Sciences Center, Lubbock, TX, USA.
Zitieren
APA
Nakaphan, P., Damara, I., Lerttiendamrong, B., Shotelersuk, V., & Chaisrimaneepan, N. (2025). Comparative evaluation of large language models on multiple-choice and image-based rheumatology questions. Rheumatology International, 46(1). https://doi.org/10.1007/s00296-025-06053-5
Vancouver
Nakaphan P, Damara I, Lerttiendamrong B, Shotelersuk V, Chaisrimaneepan N. Comparative evaluation of large language models on multiple-choice and image-based rheumatology questions. Rheumatology International. 2025;46(1). doi:10.1007/s00296-025-06053-5
BibTeX
@article{Nakaphan2025,
author = {Nakaphan, Pannathorn and Damara, Ivan and Lerttiendamrong, Bhoowit and Shotelersuk, Varote and Chaisrimaneepan, Nattanicha},
title = {Comparative evaluation of large language models on multiple-choice and image-based rheumatology questions},
journal = {Rheumatology International},
year = {2025},
volume = {46},
number = {1},
doi = {10.1007/s00296-025-06053-5}
}
EUVIMED