EUVIMEDEUVIMED entdecken
VergleichsstudieOpen Access

Measuring the Accuracy and Reproducibility of DeepSeek R1, Claude 3.5 Sonnet, and GPT-4.1 on Complex Clinical Scenarios

Robert E Hoyt, Maria Bajwa

Applied Clinical Informatics · 01.01.2026 · 17 (01) · 064-072

DOI: 10.1055/a-2807-4256 ↗

Kurz gefasst

Vergleichsstudie in der Zeitschrift Applied Clinical Informatics, erschienen am 01.01.2026. Verfasst von Robert E Hoyt und 1 weiteren Autor:innen. Beteiligte Einrichtung: Internal Medicine, Virginia Commonwealth University, Richmond, Virginia, United States. Hauptthemen laut MEDLINE: Large Language Models, Medical Informatics. Die Arbeit wurde bisher 1-mal zitiert (Crossref) und nennt 26 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

The integration of large language models (LLMs) into clinical diagnostics presents significant challenges regarding their accuracy and reliability. This study aimed to evaluate the performance of DeepSeek R1, an open-source reasoning model, alongside two other LLMs, GPT-4.1 and Claude 3.5 Sonnet, across multiple-choice clinical cases. A dataset of complex medical cases representative of real-world clinical practice was selected. For efficiency, models were accessed via application programming interfaces (APIs) and assessed using standardized prompts and a predefined evaluation protocol. The models demonstrated an overall accuracy of 77.1%, with GPT-4 producing the fewest errors and Claude 3.5 the most. The reproducibility analysis indicated that the tests were very repeatable: DeepSeek (100%), GPT-4.1 (97.5%), and Claude 3.5 Sonnet (92%). While LLMs show promise for enhancing diagnostics, ongoing scrutiny is required to address error rates and validate standard medical answers. Given the limited dataset and prompting protocol, findings should not be interpreted as broader equivalence in real-world clinical reasoning. This study demonstrates the need for robust evaluation standards, attention to error rates, and further research.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Large Language ModelsMedical InformaticsHumansReproducibility of Results

Autor:innen und Institutionen

  • Internal Medicine, Virginia Commonwealth University, Richmond, Virginia, United States.
  • Department of Health Professions Education (HPEd), School of Health & Rehabilitation Sciences, MGH Institute of Health Professions, Boston, Massachusetts, United States.

Zitieren

APA

Hoyt, R. E., & Bajwa, M. (2026). Measuring the Accuracy and Reproducibility of DeepSeek R1, Claude 3.5 Sonnet, and GPT-4.1 on Complex Clinical Scenarios. Applied Clinical Informatics, 17(01), 064-072. https://doi.org/10.1055/a-2807-4256

Vancouver

Hoyt RE, Bajwa M. Measuring the Accuracy and Reproducibility of DeepSeek R1, Claude 3.5 Sonnet, and GPT-4.1 on Complex Clinical Scenarios. Applied Clinical Informatics. 2026;17(01):064-072. doi:10.1055/a-2807-4256

BibTeX
@article{Hoyt2026,
  author = {Hoyt, Robert E and Bajwa, Maria},
  title = {Measuring the Accuracy and Reproducibility of DeepSeek R1, Claude 3.5 Sonnet, and GPT-4.1 on Complex Clinical Scenarios},
  journal = {Applied Clinical Informatics},
  year = {2026},
  volume = {17},
  number = {01},
  pages = {064--072},
  doi = {10.1055/a-2807-4256}
}
Kostenlos starten →