EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Assessing the Accuracy of Diagnostic Capabilities of Large Language Models

Andrada Elena Urda-Cîmpean iD, Daniel-Corneliu Leucuța iD, Cristina Drugan iD, Alina-Gabriela Duțu, Tudor Călinici iD, Tudor Drugan iD

Diagnostics · 29.06.2025 · 15 (13) · 1657

DOI: 10.3390/diagnostics15131657 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Diagnostics, online erschienen am 29.06.2025. Verfasst von Andrada Elena Urda-Cîmpean und 5 weiteren Autor:innen. Beteiligte Einrichtung: Department of Medical Informatics and Biostatistics, Iuliu Hațieganu University of Medicine and Pharmacy, 400349 Cluj-Napoca, Romania. Die Arbeit wurde bisher 16-mal zitiert (Crossref) und nennt 29 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Iuliu Hațieganu University of Medicine and Pharmacy, “Iuliu Hațieganu” University of Medicine and Pharmacy Cluj-Napoca, Romania.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

In recent years, numerous artificial intelligence applications, especially generative large language models, have evolved in the medical field. This study conducted a structured comparative analysis of four leading generative large language models (LLMs)—ChatGPT-4o (OpenAI), Grok-3 (xAI), Gemini-2.0 Flash (Google), and DeepSeek-V3 (DeepSeek)—to evaluate their diagnostic performance in clinical case scenarios.

Methods

We assessed medical knowledge recall and clinical reasoning capabilities through staged, progressively complex cases, with responses graded by expert raters using a 0–5 scale.

Results

All models performed better on knowledge-based questions than on reasoning tasks, highlighting the ongoing limitations in contextual diagnostic synthesis. Overall, DeepSeek outperformed the other models, achieving significantly higher scores across all evaluation dimensions (p < 0.05), particularly in regards to medical reasoning tasks.

Conclusions

While these findings support the feasibility of using LLMs for medical training and decision support, the study emphasizes the need for improved interpretability, prompt optimization, and rigorous benchmarking to ensure clinical reliability. This structured, comparative approach contributes to ongoing efforts to establish standardized evaluation frameworks for integrating LLMs into diagnostic workflows.

Schlagwörter

Stichwörter der Autor:innen

artificial intelligencediagnostic accuracylarge language modesmedical education

Autor:innen und Institutionen

  • Department of Medical Informatics and Biostatistics, Iuliu Hațieganu University of Medicine and Pharmacy, 400349 Cluj-Napoca, Romania.
  • Department of Medical Biochemistry, Iuliu Hațieganu University of Medicine and Pharmacy, 400349 Cluj-Napoca, Romania.

ORCID-Profile: Andrada Elena Urda-Cîmpean, Daniel-Corneliu Leucuța, Cristina Drugan, Tudor Călinici, Tudor Drugan

Förderung

  • Iuliu Hațieganu University of Medicine and Pharmacy
  • “Iuliu Hațieganu” University of Medicine and Pharmacy Cluj-Napoca, Romania

Zitieren

APA

Urda-Cîmpean, A. E., Leucuța, D. C., Drugan, C., Duțu, A. G., Călinici, T., & Drugan, T. (2025). Assessing the Accuracy of Diagnostic Capabilities of Large Language Models. Diagnostics, 15(13), 1657. https://doi.org/10.3390/diagnostics15131657

Vancouver

Urda-Cîmpean AE, Leucuța DC, Drugan C, Duțu AG, Călinici T, Drugan T. Assessing the Accuracy of Diagnostic Capabilities of Large Language Models. Diagnostics. 2025;15(13):1657. doi:10.3390/diagnostics15131657

BibTeX
@article{UrdaCmpean2025,
  author = {Urda-Cîmpean, Andrada Elena and Leucuța, Daniel-Corneliu and Drugan, Cristina and Duțu, Alina-Gabriela and Călinici, Tudor and Drugan, Tudor},
  title = {Assessing the Accuracy of Diagnostic Capabilities of Large Language Models},
  journal = {Diagnostics},
  year = {2025},
  volume = {15},
  number = {13},
  pages = {1657},
  doi = {10.3390/diagnostics15131657}
}
Kostenlos starten →