Evaluating the Applicability of Advanced Large Language Models in Laboratory Medicine Test Questions: A Comparative Performance Study
Advances in Medical Education and Practice · 2026-02 · Volume 17 · 1-13
DOI: 10.2147/amep.s576651 ↗Kurz gefasst
Fachartikel in der Zeitschrift Advances in Medical Education and Practice, 2026-02. Verfasst von Wenzheng Han und 12 weiteren Autor:innen. Beteiligte Einrichtung: Department of Clinical Laboratory, The First Affiliated Hospital, Wannan Medical College, Wuhu, Anhui, People's Republic of China. Die Arbeit nennt 21 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-NC 3.0).
Abstract (Originalsprache)
Background
While large language models (LLMs) show promise in medical education, their comprehensive performance in specialized domains like medical laboratory science remains inadequately assessed.
Purpose
This study aimed to evaluate advanced LLMs on medical laboratory questions, assessing accuracy, natural language generation (NLG) quality, reasoning performance, and efficiency.
Methods
We conducted a multi-faceted evaluation of three advanced LLMs (DeepSeek-R1, Gemini-2.5 Pro, GPT-5), benchmarking them against medical laboratory scientists and earlier ChatGPT versions. The evaluation utilized 493 questions sourced from the internal Medical Laboratory Test Bank of Wannan Medical College. These questions comprised both knowledge-based and reasoning-based single- and multiple-choice types (SCQs and MCQs). Performance was measured by accuracy, Macro-F1, response time, NLG scores (ROUGE-L, METEOR), and structured logical reasoning assessment. Appropriate statistical tests (including χ2, Wilcoxon, ANOVA, and non-parametric alternatives) with post-hoc corrections were applied to determine significance.
Results
DeepSeek-R1's accuracy on total questions was 78.3%, nearing the 79.3% of the higher-performing senior expert. Notably, it excelled at complex reasoning-based MCQ, demonstrating an advantage over senior experts with an accuracy of 64.4%, compared to 58.7% (SMLS-1) and 56.7% (SMLS-2). While ChatGPT-5 was the fastest model, DeepSeek-R1 exhibited intermediate efficiency, aligning with human experts on SCQ but requiring more time for MCQ. In terms of NLG, DeepSeek-R1 consistently achieved the highest scores, with ROUGE-L scores of 0.36 ± 0.14 (Total Q), 0.33 ± 0.15 (SCQ), and 0.38 ± 0.13 (MCQ), and METEOR scores of 0.53 ± 0.19 (Total Q), 0.40 ± 0.17 (SCQ), and 0.63 ± 0.14 (MCQ). Furthermore, it significantly outperformed all other LLMs in logical reasoning comprehensiveness. A critical strength was its consistent integration of key negative findings, vital for diagnosis.
Conclusion
DeepSeek-R1 approaches or even surpasses senior expert performance in certain tasks, showing strong potential as an effective tool for education and assessment despite slower processing times.
Schlagwörter
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Department of Clinical Laboratory, The First Affiliated Hospital, Wannan Medical College, Wuhu, Anhui, People's Republic of China.
- Department of Pediatrics, The First Affiliated Hospital, Wannan Medical College, Wuhu, Anhui, People's Republic of China.
- School of Laboratory Medicine, Wannan Medical College, Wuhu, Anhui, People's Republic of China.
- Department of Infectious Diseases, The First Affiliated Hospital, Wannan Medical College, Wuhu, Anhui, People's Republic of China.
- Department of Eye Center, The Second Affiliated Hospital, School of Medicine, Zhejiang University, Hangzhou, Zhejiang, People's Republic of China.
- Department of Nuclear Medicine, The First Affiliated Hospital, Wannan Medical College, Wuhu, Anhui, People's Republic of China.
ORCID-Profile: Gang Feng, Jianghua Yang
Zitieren
APA
Han, W., Zhu, W., Feng, G., Wang, Y., Chen, G., Zhou, H., Quan, B., Wu, Q., Yang, J., Jin, K., Tao, S., Li, X., & Chen, Q. (2026). Evaluating the Applicability of Advanced Large Language Models in Laboratory Medicine Test Questions: A Comparative Performance Study. Advances in Medical Education and Practice, Volume 17, 1-13. https://doi.org/10.2147/amep.s576651
Vancouver
Han W, Zhu W, Feng G, Wang Y, Chen G, Zhou H, et al. Evaluating the Applicability of Advanced Large Language Models in Laboratory Medicine Test Questions: A Comparative Performance Study. Advances in Medical Education and Practice. 2026;Volume 17:1-13. doi:10.2147/amep.s576651
BibTeX
@article{Han2026,
author = {Han, Wenzheng and Zhu, Wenkai and Feng, Gang and Wang, Yankang and Chen, Guang and Zhou, Huan and Quan, Bin and Wu, Qiwen and Yang, Jianghua and Jin, Kai and Tao, Shaoneng and Li, Xiaoning and Chen, Qing},
title = {Evaluating the Applicability of Advanced Large Language Models in Laboratory Medicine Test Questions: A Comparative Performance Study},
journal = {Advances in Medical Education and Practice},
year = {2026},
volume = {Volume 17},
pages = {1--13},
doi = {10.2147/amep.s576651}
}
EUVIMED