Comparative analysis of large language models in clinical diagnosis: performance evaluation across common and complex medical cases
JAMIA Open · 02.05.2025 · 8 (3)
DOI: 10.1093/jamiaopen/ooaf055 ↗Kurz gefasst
Fachartikel in der Zeitschrift JAMIA Open, online erschienen am 12.06.2025. Verfasst von Mehmed T Dinc und 3 weiteren Autor:innen. Beteiligte Einrichtung: Department of Medicine, Boston Medical Center, Boston, MA 02118, United States. Die Arbeit wurde bisher 44-mal zitiert (Crossref) und nennt 22 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
Objectives
This study aimed to systematically evaluate and compare the diagnostic performance of leading large language models (LLMs) in common and complex clinical scenarios, assessing their potential for enhancing clinical reasoning and diagnostic accuracy in authentic clinical decision-making processes. Materials and Methods Diagnostic capabilities of advanced LLMs (Anthropic’s Claude, OpenAI’s GPT variants, Google’s Gemini) were assessed using 60 common cases and 104 complex, real-world cases from Clinical Problem Solvers’ morning rounds. Clinical details were disclosed in stages, mirroring authentic clinical decision-making. Models were evaluated on primary and differential diagnosis accuracy at each stage.
Results
Advanced LLMs showed high diagnostic accuracy (>90%) in common scenarios, with Claude 3.7 achieving perfect accuracy (100%) in certain conditions. In complex cases, Claude 3.7 achieved the highest accuracy (83.3%) at the final diagnostic stage, significantly outperforming smaller models. Smaller models notably performed well in common scenarios, matching the performance of larger models.
Discussion
This study evaluated leading LLMs for diagnostic accuracy using staged information disclosure, mirroring real-world practice. Notably, Claude 3.7 Sonnet was the top performer. Employing a novel LLM-based evaluation method for large-scale analysis, the research highlights artificial intelligence’s (AI’s) potential to enhance diagnostics. It underscores the need for useful frameworks to translate accuracy into clinical impact and integrate AI into medical education.
Conclusion
Leading LLMs show remarkable diagnostic accuracy in diverse clinical cases. To fully realize their potential for improving patient care, we must now focus on creating practical implementation frameworks and translational research to integrate these powerful AI tools into medicine.
Schlagwörter
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Department of Medicine, Boston Medical Center, Boston, MA 02118, United States.
- Department of Medicine, St Elizabeth's Medical Center, Boston, MA 02135, United States.
- Department of Medicine, Mount Auburn Hospital, Harvard Medical School, Cambridge, MA 02138, United States.
ORCID-Profile: Mehmed T Dinc
Zitieren
APA
Dinc, M. T., Bardak, A. E., Bahar, F., & Noronha, C. (2025). Comparative analysis of large language models in clinical diagnosis: performance evaluation across common and complex medical cases. JAMIA Open, 8(3). https://doi.org/10.1093/jamiaopen/ooaf055
Vancouver
Dinc MT, Bardak AE, Bahar F, Noronha C. Comparative analysis of large language models in clinical diagnosis: performance evaluation across common and complex medical cases. JAMIA Open. 2025;8(3). doi:10.1093/jamiaopen/ooaf055
BibTeX
@article{Dinc2025,
author = {Dinc, Mehmed T and Bardak, Ali E and Bahar, Furkan and Noronha, Craig},
title = {Comparative analysis of large language models in clinical diagnosis: performance evaluation across common and complex medical cases},
journal = {JAMIA Open},
year = {2025},
volume = {8},
number = {3},
doi = {10.1093/jamiaopen/ooaf055}
}
EUVIMED