Evaluating Large‐Scale and Lightweight Large Language Models for Traditional Chinese Medicine Exam Questions: A Comparative Study
Journal of Evidence-Based Medicine · 2026-03 · 19 (1)
DOI: 10.1111/jebm.70118 ↗Kurz gefasst
Vergleichsstudie in der Zeitschrift Journal of Evidence-Based Medicine, online erschienen am 11.02.2026. Verfasst von Yizhen Li und 8 weiteren Autor:innen. Beteiligte Einrichtung: The First Affiliated Hospital of Henan University of Chinese Medicine, Zhengzhou, China. Hauptthemen laut MEDLINE: Large Language Models, Medicine, Chinese Traditional, Educational Measurement. Die Arbeit wurde bisher 3-mal zitiert (Crossref) und nennt 41 Literaturangaben. Gefördert von: Henan Provincial Natural Science Foundation, Henan Provincial Health Commission National Traditional Chinese Medicine Inheritance and Innovation Center.
Abstract (Originalsprache)
Background
Traditional Chinese medicine (TCM) with knowledge‐intensive framework poses unique challenges to performance for large language models (LLMs). Although TCM‐specific benchmarks and models have been developed, the performance of lightweight LLMs remains insufficiently investigated. This study presents a systematic evaluation and comparison of large‐scale and lightweight LLMs to assess their capabilities and deployment trade‐offs.
Methods
We developed TCM‐related question‐answering, a dataset comprising 801 questions derived from TCM textbooks. Eleven LLMs were evaluated under zero‐shot and few‐shot prompting conditions in both English and Chinese. Performance was primarily measured by accuracy.
Results
Large‐scale LLMs achieved high accuracy on single‐choice (69.01%–90.92%) and true/false questions (52.34%–59.38%) but performed poorly on multiple‐choice questions, with a maximum accuracy of only 8.40%. Lightweight LLMs (2.10%–49.48%) generally lagged behind larger LLMs (6.30%–95.07%). However, Qwen3‐1.7B (5.92%–54.20%) stood out and even surpassed the domain‐specialized TCMChat‐7B (2.10%–36.98%). Few‐shot prompting enhanced performance in 8/11 (72.7%) of the models, Chinese prompts yielded better results than English in 9/11 (81.8%) of the models. Symptomatic diagnosis emerged as the most challenging reasoning category across all models (16.75%–48.07%).
Conclusion
This study demonstrates that although large‐scale LLMs exhibit strong knowledge recall in TCM, their suboptimal performance on multiple‐choice questions and substantial computational costs may limit their practical applicability in clinical settings. The robust performance of Qwen3‐1.7B indicates that effective model optimization and domain‐specific training may offer greater advantages than simply increasing model size. While the current evaluation is based on examination‐style tasks and does not involve real‐world clinical decision‐making, our findings provide insights to support the deployment of optimized models in resource‐constrained healthcare environments.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- The First Affiliated Hospital of Henan University of Chinese Medicine, Zhengzhou, China.
- School of Computer Science and Engineering, Beihang University, Beijing, China.
- School of Life and Science, Beijing University of Chinese Medicine, Beijing, China.
Förderung
- Henan Provincial Natural Science Foundation
- Henan Provincial Health Commission National Traditional Chinese Medicine Inheritance and Innovation Center
Zitieren
APA
Li, Y., Huang, S., Qi, J., Lu, Y., Quan, L., Han, D., Li, B., Liu, X., & Luan, Z. (2026). Evaluating Large‐Scale and Lightweight Large Language Models for Traditional Chinese Medicine Exam Questions: A Comparative Study. Journal of Evidence-Based Medicine, 19(1). https://doi.org/10.1111/jebm.70118
Vancouver
Li Y, Huang S, Qi J, Lu Y, Quan L, Han D, et al. Evaluating Large‐Scale and Lightweight Large Language Models for Traditional Chinese Medicine Exam Questions: A Comparative Study. Journal of Evidence-Based Medicine. 2026;19(1). doi:10.1111/jebm.70118
BibTeX
@article{Li2026,
author = {Li, Yizhen and Huang, Shaohan and Qi, Jiaxing and Lu, Yao and Quan, Lei and Han, Dongran and Li, Bin and Liu, Xincan and Luan, Zhongzhi},
title = {Evaluating Large‐Scale and Lightweight Large Language Models for Traditional Chinese Medicine Exam Questions: A Comparative Study},
journal = {Journal of Evidence-Based Medicine},
year = {2026},
volume = {19},
number = {1},
doi = {10.1111/jebm.70118}
}
EUVIMED