Measuring the Accuracy and Reproducibility of DeepSeek R1, Claude 3.5 Sonnet, and GPT-4.1 on Complex Clinical Scenarios
Applied Clinical Informatics · 01.01.2026 · 17 (01) · 064-072
DOI: 10.1055/a-2807-4256 ↗Kurz gefasst
Vergleichsstudie in der Zeitschrift Applied Clinical Informatics, erschienen am 01.01.2026. Verfasst von Robert E Hoyt und 1 weiteren Autor:innen. Beteiligte Einrichtung: Internal Medicine, Virginia Commonwealth University, Richmond, Virginia, United States. Hauptthemen laut MEDLINE: Large Language Models, Medical Informatics. Die Arbeit wurde bisher 1-mal zitiert (Crossref) und nennt 26 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.
Abstract (Originalsprache)
The integration of large language models (LLMs) into clinical diagnostics presents significant challenges regarding their accuracy and reliability. This study aimed to evaluate the performance of DeepSeek R1, an open-source reasoning model, alongside two other LLMs, GPT-4.1 and Claude 3.5 Sonnet, across multiple-choice clinical cases. A dataset of complex medical cases representative of real-world clinical practice was selected. For efficiency, models were accessed via application programming interfaces (APIs) and assessed using standardized prompts and a predefined evaluation protocol. The models demonstrated an overall accuracy of 77.1%, with GPT-4 producing the fewest errors and Claude 3.5 the most. The reproducibility analysis indicated that the tests were very repeatable: DeepSeek (100%), GPT-4.1 (97.5%), and Claude 3.5 Sonnet (92%). While LLMs show promise for enhancing diagnostics, ongoing scrutiny is required to address error rates and validate standard medical answers. Given the limited dataset and prompting protocol, findings should not be interpreted as broader equivalence in real-world clinical reasoning. This study demonstrates the need for robust evaluation standards, attention to error rates, and further research.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Autor:innen und Institutionen
- Internal Medicine, Virginia Commonwealth University, Richmond, Virginia, United States.
- Department of Health Professions Education (HPEd), School of Health & Rehabilitation Sciences, MGH Institute of Health Professions, Boston, Massachusetts, United States.
Zitieren
APA
Hoyt, R. E., & Bajwa, M. (2026). Measuring the Accuracy and Reproducibility of DeepSeek R1, Claude 3.5 Sonnet, and GPT-4.1 on Complex Clinical Scenarios. Applied Clinical Informatics, 17(01), 064-072. https://doi.org/10.1055/a-2807-4256
Vancouver
Hoyt RE, Bajwa M. Measuring the Accuracy and Reproducibility of DeepSeek R1, Claude 3.5 Sonnet, and GPT-4.1 on Complex Clinical Scenarios. Applied Clinical Informatics. 2026;17(01):064-072. doi:10.1055/a-2807-4256
BibTeX
@article{Hoyt2026,
author = {Hoyt, Robert E and Bajwa, Maria},
title = {Measuring the Accuracy and Reproducibility of DeepSeek R1, Claude 3.5 Sonnet, and GPT-4.1 on Complex Clinical Scenarios},
journal = {Applied Clinical Informatics},
year = {2026},
volume = {17},
number = {01},
pages = {064--072},
doi = {10.1055/a-2807-4256}
}
EUVIMED