Stepwise Diagnostic Evaluation of Chinese Large Language Models: Comparative Study of Common and Rare Diseases
Journal of Medical Internet Research · 06.08.2026 · 28 · e89963-e89963
DOI: 10.2196/89963 ↗Kurz gefasst
Vergleichsstudie in der Zeitschrift Journal of Medical Internet Research, online erschienen am 06.08.2026. Verfasst von Jiayi Wang und 2 weiteren Autor:innen. Beteiligte Einrichtung: Department of Health Management and Policy, School of Public Health, Capital Medical University, No. 10 Xitoutiao, Youanmenwai, Fengtai District, Bei…. Hauptthemen laut MEDLINE: Large Language Models, Pulmonary Disease, Chronic Obstructive, Rare Diseases. Die Arbeit nennt 43 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.
Abstract (Originalsprache)
Background
Large language models (LLMs) are increasingly applied in clinical decision support, yet their diagnostic performance in Chinese-language settings and under realistic clinical workflows remains unclear. In particular, how LLMs perform across diseases with different prevalence and under stepwise diagnostic processes has not been well characterized.
Objective
This study aimed to evaluate the diagnostic capabilities of LLMs for common diseases and rare diseases using clinical vignettes within a hypothetico-deductive framework and to identify their potential and limitations for clinical diagnosis.
Methods
We evaluated 4 Chinese LLMs (Doubao 1.5, DeepSeek-V3, Kimi K1.5, and Leftdoctor GPT 3.5) using 56 clinical cases (28 chronic obstructive pulmonary disease [COPD], and 28 relapsing polychondritis [RP]) sourced from the China Clinical Case Results Database (March 31-April 14, 2025). Patient information was provided incrementally, starting with the initial medical history, followed by physical examination, and laboratory results. Evaluation metrics included top-3 accuracy (RTop3D), top-1 accuracy (RTopD), final diagnostic accuracy (RFA), and mean reciprocal rank (MRR). Statistical analysis was performed using generalized estimating equations (GEE), Friedman tests, and Wilcoxon signed-rank tests with Bonferroni correction. In addition, a qualitative analysis was conducted to characterize recurrent patterns of diagnostic errors.
Results
LLMs demonstrated significantly higher diagnostic accuracy for COPD compared to RP across all metrics ( P
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Department of Health Management and Policy, School of Public Health, Capital Medical University, No. 10 Xitoutiao, Youanmenwai, Fengtai District, Beijing, Beijing, 100069, China, 86 010 83911303.
Zitieren
APA
Wang, J., Yang, J., & Guo, R. (2026). Stepwise Diagnostic Evaluation of Chinese Large Language Models: Comparative Study of Common and Rare Diseases. Journal of Medical Internet Research, 28, e89963-e89963. https://doi.org/10.2196/89963
Vancouver
Wang J, Yang J, Guo R. Stepwise Diagnostic Evaluation of Chinese Large Language Models: Comparative Study of Common and Rare Diseases. Journal of Medical Internet Research. 2026;28:e89963-e89963. doi:10.2196/89963
BibTeX
@article{Wang2026,
author = {Wang, Jiayi and Yang, Jiao and Guo, Rui},
title = {Stepwise Diagnostic Evaluation of Chinese Large Language Models: Comparative Study of Common and Rare Diseases},
journal = {Journal of Medical Internet Research},
year = {2026},
volume = {28},
pages = {e89963--e89963},
doi = {10.2196/89963}
}
EUVIMED