EUVIMEDOpdag EUVIMED
VergleichsstudieOpen access

Stepwise Diagnostic Evaluation of Chinese Large Language Models: Comparative Study of Common and Rare Diseases

Jiayi Wang, Jiao Yang, Rui Guo

Journal of Medical Internet Research · 2026-08-06 · 28 · e89963-e89963

DOI: 10.2196/89963 ↗

At a glance

Vergleichsstudie in Journal of Medical Internet Research, published online on 2026-08-06. Written by Jiayi Wang and 2 co-authors. Institution: Department of Health Management and Policy, School of Public Health, Capital Medical University, No. 10 Xitoutiao, Youanmenwai, Fengtai District, Bei…. Major MEDLINE topics: Large Language Models, Pulmonary Disease, Chronic Obstructive, Rare Diseases. 43 references. The full text is freely available in PubMed Central.

AbstractSubjectsBibliografiske oplysningerCiteRelated studies

Abstract (original language)

Background

Large language models (LLMs) are increasingly applied in clinical decision support, yet their diagnostic performance in Chinese-language settings and under realistic clinical workflows remains unclear. In particular, how LLMs perform across diseases with different prevalence and under stepwise diagnostic processes has not been well characterized.

Objective

This study aimed to evaluate the diagnostic capabilities of LLMs for common diseases and rare diseases using clinical vignettes within a hypothetico-deductive framework and to identify their potential and limitations for clinical diagnosis.

Methods

We evaluated 4 Chinese LLMs (Doubao 1.5, DeepSeek-V3, Kimi K1.5, and Leftdoctor GPT 3.5) using 56 clinical cases (28 chronic obstructive pulmonary disease [COPD], and 28 relapsing polychondritis [RP]) sourced from the China Clinical Case Results Database (March 31-April 14, 2025). Patient information was provided incrementally, starting with the initial medical history, followed by physical examination, and laboratory results. Evaluation metrics included top-3 accuracy (RTop3D), top-1 accuracy (RTopD), final diagnostic accuracy (RFA), and mean reciprocal rank (MRR). Statistical analysis was performed using generalized estimating equations (GEE), Friedman tests, and Wilcoxon signed-rank tests with Bonferroni correction. In addition, a qualitative analysis was conducted to characterize recurrent patterns of diagnostic errors.

Results

LLMs demonstrated significantly higher diagnostic accuracy for COPD compared to RP across all metrics ( P

Subjects

MeSH terms (MEDLINE)

Large Language ModelsPulmonary Disease, Chronic ObstructiveRare DiseasesHumansChina

Author keywords

artificial intelligenceclinical decision-makingdecision support systemsdiagnostic accuracylarge language models

Authors and institutions

  • Department of Health Management and Policy, School of Public Health, Capital Medical University, No. 10 Xitoutiao, Youanmenwai, Fengtai District, Beijing, Beijing, 100069, China, 86 010 83911303.

Cite

APA

Wang, J., Yang, J., & Guo, R. (2026). Stepwise Diagnostic Evaluation of Chinese Large Language Models: Comparative Study of Common and Rare Diseases. Journal of Medical Internet Research, 28, e89963-e89963. https://doi.org/10.2196/89963

Vancouver

Wang J, Yang J, Guo R. Stepwise Diagnostic Evaluation of Chinese Large Language Models: Comparative Study of Common and Rare Diseases. Journal of Medical Internet Research. 2026;28:e89963-e89963. doi:10.2196/89963

BibTeX
@article{Wang2026,
  author = {Wang, Jiayi and Yang, Jiao and Guo, Rui},
  title = {Stepwise Diagnostic Evaluation of Chinese Large Language Models: Comparative Study of Common and Rare Diseases},
  journal = {Journal of Medical Internet Research},
  year = {2026},
  volume = {28},
  pages = {e89963--e89963},
  doi = {10.2196/89963}
}
Start for free →