EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Comparative analysis of large language models in clinical diagnosis: performance evaluation across common and complex medical cases

Mehmed T Dinc iD, Ali E Bardak, Furkan Bahar, Craig Noronha

JAMIA Open · 02.05.2025 · 8 (3)

DOI: 10.1093/jamiaopen/ooaf055 ↗

Kurz gefasst

Fachartikel in der Zeitschrift JAMIA Open, online erschienen am 12.06.2025. Verfasst von Mehmed T Dinc und 3 weiteren Autor:innen. Beteiligte Einrichtung: Department of Medicine, Boston Medical Center, Boston, MA 02118, United States. Die Arbeit wurde bisher 44-mal zitiert (Crossref) und nennt 22 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Objectives

This study aimed to systematically evaluate and compare the diagnostic performance of leading large language models (LLMs) in common and complex clinical scenarios, assessing their potential for enhancing clinical reasoning and diagnostic accuracy in authentic clinical decision-making processes. Materials and Methods Diagnostic capabilities of advanced LLMs (Anthropic’s Claude, OpenAI’s GPT variants, Google’s Gemini) were assessed using 60 common cases and 104 complex, real-world cases from Clinical Problem Solvers’ morning rounds. Clinical details were disclosed in stages, mirroring authentic clinical decision-making. Models were evaluated on primary and differential diagnosis accuracy at each stage.

Results

Advanced LLMs showed high diagnostic accuracy (>90%) in common scenarios, with Claude 3.7 achieving perfect accuracy (100%) in certain conditions. In complex cases, Claude 3.7 achieved the highest accuracy (83.3%) at the final diagnostic stage, significantly outperforming smaller models. Smaller models notably performed well in common scenarios, matching the performance of larger models.

Discussion

This study evaluated leading LLMs for diagnostic accuracy using staged information disclosure, mirroring real-world practice. Notably, Claude 3.7 Sonnet was the top performer. Employing a novel LLM-based evaluation method for large-scale analysis, the research highlights artificial intelligence’s (AI’s) potential to enhance diagnostics. It underscores the need for useful frameworks to translate accuracy into clinical impact and integrate AI into medical education.

Conclusion

Leading LLMs show remarkable diagnostic accuracy in diverse clinical cases. To fully realize their potential for improving patient care, we must now focus on creating practical implementation frameworks and translational research to integrate these powerful AI tools into medicine.

Schlagwörter

Stichwörter der Autor:innen

artificial intelligenceclinical decision supportlarge language modelsmedical diagnosis

Autor:innen und Institutionen

  • Department of Medicine, Boston Medical Center, Boston, MA 02118, United States.
  • Department of Medicine, St Elizabeth's Medical Center, Boston, MA 02135, United States.
  • Department of Medicine, Mount Auburn Hospital, Harvard Medical School, Cambridge, MA 02138, United States.

ORCID-Profile: Mehmed T Dinc

Zitieren

APA

Dinc, M. T., Bardak, A. E., Bahar, F., & Noronha, C. (2025). Comparative analysis of large language models in clinical diagnosis: performance evaluation across common and complex medical cases. JAMIA Open, 8(3). https://doi.org/10.1093/jamiaopen/ooaf055

Vancouver

Dinc MT, Bardak AE, Bahar F, Noronha C. Comparative analysis of large language models in clinical diagnosis: performance evaluation across common and complex medical cases. JAMIA Open. 2025;8(3). doi:10.1093/jamiaopen/ooaf055

BibTeX
@article{Dinc2025,
  author = {Dinc, Mehmed T and Bardak, Ali E and Bahar, Furkan and Noronha, Craig},
  title = {Comparative analysis of large language models in clinical diagnosis: performance evaluation across common and complex medical cases},
  journal = {JAMIA Open},
  year = {2025},
  volume = {8},
  number = {3},
  doi = {10.1093/jamiaopen/ooaf055}
}
Kostenlos starten →