EUVIMEDEUVIMED entdecken
Crossref

Evaluation of DeepSeek-R1 and contemporary large language models on the radiology board examination: A milestone achieved as open-source model matches performance with closed-source model

Takeshi Nakaura, Naoki Kobayashi, Takanori Masuda, Yasunori Nagayama, Hiroyuki Uetani, Masafumi Kidoh, Seitaro Oda, Yoshinori Funama, Toshinori Hirai

European Journal of Radiology · 2026

DOI: 10.1016/j.ejrad.2025.112608 ↗

Kurz gefasst

Fachartikel in der Zeitschrift European Journal of Radiology, 2026. Verfasst von Takeshi Nakaura und 8 weiteren Autor:innen. Beteiligte Einrichtung: Department of Central Radiology, Kumamoto University Hospital, 1-1-1 Honjo, Kumamoto 860-8556, Japan. Electronic address: kff00712@nifty.com. Hauptthemen laut MEDLINE: Radiology, Educational Measurement, Specialty Boards, Clinical Competence. Die Arbeit wurde bisher 1-mal zitiert (Crossref) und nennt 24 Literaturangaben.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

OBJECTIVES: Recent advances in large language models (LLMs), especially reasoning LLMs have demonstrated impressive reasoning capabilities in specialized domains. The purpose of this study is to evaluate the performance of new open-source reasoning LLM, DeepSeek-R1 and other contemporary LLMs on radiology board examination questions, comparing their accuracy to human radiologists. MATERIALS AND METHODS: We assessed 10 LLMs, including both closed-source models (GPT-4o, o1, o3-mini, Claude 3.5 Sonnet, Gemini Flash 2.0) and open-source models (DeepSeek-R1, its distilled versions, and Llama 3.3 70B), on 105 non-image multiple-choice questions from the 2024 official board examination of the Japan Radiological Society. We evaluated accuracies of LLMs and compared those to accuracies of examinees (3rd-year radiology residents). RESULTS: DeepSeek-R1 and OpenAI o1 each answered 92/105 items correctly (87.6 %, 95 % CI: 80.1-92.5); McNemar's test showed no significant head-to-head difference (p = 1.00), outperforming human radiologists' mean accuracy of 67.6 ± 9.3 % (z-score = 2.15, equivalent to 98.4th percentile). DeepSeek-R1 demonstrated superior cost-efficiency with API costs approximately 1/27th of o1. DeepSeek-R1's distilled 32B model (61.9 % accuracy) outperformed Llama 3.3 70B (57.1 %), despite having fewer parameters. CONCLUSION: Open-source DeepSeek-R1 matches the performance of top closed-source models while offering superior cost-efficiency in radiology knowledge assessment.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

RadiologyEducational MeasurementSpecialty BoardsClinical CompetenceCertificationHumansJapanLarge Language Models

Stichwörter der Autor:innen

Board examinationCost-efficiencyLarge language modelsOpen-sourceRadiology education

Autor:innen und Institutionen

  • Department of Central Radiology, Kumamoto University Hospital, 1-1-1 Honjo, Kumamoto 860-8556, Japan. Electronic address: kff00712@nifty.com.
  • Department of Central Radiology, Kumamoto University Hospital, 1-1-1 Honjo, Kumamoto 860-8556, Japan.
  • Department of Radiological Technology, Faculty of Health Science and Technology, Kawasaki University of Medical Welfare, Okayama, Japan.
  • Department of Medical Physics, Faculty of Life Sciences, Kumamoto University, Kumamoto, Japan.

Zitieren

APA

Nakaura, T., Kobayashi, N., Masuda, T., Nagayama, Y., Uetani, H., Kidoh, M., Oda, S., Funama, Y., & Hirai, T. (2026). Evaluation of DeepSeek-R1 and contemporary large language models on the radiology board examination: A milestone achieved as open-source model matches performance with closed-source model. European Journal of Radiology. https://doi.org/10.1016/j.ejrad.2025.112608

Vancouver

Nakaura T, Kobayashi N, Masuda T, Nagayama Y, Uetani H, Kidoh M, et al. Evaluation of DeepSeek-R1 and contemporary large language models on the radiology board examination: A milestone achieved as open-source model matches performance with closed-source model. European Journal of Radiology. 2026. doi:10.1016/j.ejrad.2025.112608

BibTeX
@article{Nakaura2026,
  author = {Nakaura, Takeshi and Kobayashi, Naoki and Masuda, Takanori and Nagayama, Yasunori and Uetani, Hiroyuki and Kidoh, Masafumi and Oda, Seitaro and Funama, Yoshinori and Hirai, Toshinori},
  title = {Evaluation of DeepSeek-R1 and contemporary large language models on the radiology board examination: A milestone achieved as open-source model matches performance with closed-source model},
  journal = {European Journal of Radiology},
  year = {2026},
  doi = {10.1016/j.ejrad.2025.112608}
}
Kostenlos starten →