EUVIMEDEUVIMED entdecken
VergleichsstudieOpen Access

Stepwise Diagnostic Evaluation of Chinese Large Language Models: Comparative Study of Common and Rare Diseases

Jiayi Wang, Jiao Yang, Rui Guo

Journal of Medical Internet Research · 06.08.2026 · 28 · e89963-e89963

DOI: 10.2196/89963 ↗

Kurz gefasst

Vergleichsstudie in der Zeitschrift Journal of Medical Internet Research, online erschienen am 06.08.2026. Verfasst von Jiayi Wang und 2 weiteren Autor:innen. Beteiligte Einrichtung: Department of Health Management and Policy, School of Public Health, Capital Medical University, No. 10 Xitoutiao, Youanmenwai, Fengtai District, Bei…. Hauptthemen laut MEDLINE: Large Language Models, Pulmonary Disease, Chronic Obstructive, Rare Diseases. Die Arbeit nennt 43 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

Large language models (LLMs) are increasingly applied in clinical decision support, yet their diagnostic performance in Chinese-language settings and under realistic clinical workflows remains unclear. In particular, how LLMs perform across diseases with different prevalence and under stepwise diagnostic processes has not been well characterized.

Objective

This study aimed to evaluate the diagnostic capabilities of LLMs for common diseases and rare diseases using clinical vignettes within a hypothetico-deductive framework and to identify their potential and limitations for clinical diagnosis.

Methods

We evaluated 4 Chinese LLMs (Doubao 1.5, DeepSeek-V3, Kimi K1.5, and Leftdoctor GPT 3.5) using 56 clinical cases (28 chronic obstructive pulmonary disease [COPD], and 28 relapsing polychondritis [RP]) sourced from the China Clinical Case Results Database (March 31-April 14, 2025). Patient information was provided incrementally, starting with the initial medical history, followed by physical examination, and laboratory results. Evaluation metrics included top-3 accuracy (RTop3D), top-1 accuracy (RTopD), final diagnostic accuracy (RFA), and mean reciprocal rank (MRR). Statistical analysis was performed using generalized estimating equations (GEE), Friedman tests, and Wilcoxon signed-rank tests with Bonferroni correction. In addition, a qualitative analysis was conducted to characterize recurrent patterns of diagnostic errors.

Results

LLMs demonstrated significantly higher diagnostic accuracy for COPD compared to RP across all metrics ( P

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Large Language ModelsPulmonary Disease, Chronic ObstructiveRare DiseasesHumansChina

Stichwörter der Autor:innen

artificial intelligenceclinical decision-makingdecision support systemsdiagnostic accuracylarge language models

Autor:innen und Institutionen

  • Department of Health Management and Policy, School of Public Health, Capital Medical University, No. 10 Xitoutiao, Youanmenwai, Fengtai District, Beijing, Beijing, 100069, China, 86 010 83911303.

Zitieren

APA

Wang, J., Yang, J., & Guo, R. (2026). Stepwise Diagnostic Evaluation of Chinese Large Language Models: Comparative Study of Common and Rare Diseases. Journal of Medical Internet Research, 28, e89963-e89963. https://doi.org/10.2196/89963

Vancouver

Wang J, Yang J, Guo R. Stepwise Diagnostic Evaluation of Chinese Large Language Models: Comparative Study of Common and Rare Diseases. Journal of Medical Internet Research. 2026;28:e89963-e89963. doi:10.2196/89963

BibTeX
@article{Wang2026,
  author = {Wang, Jiayi and Yang, Jiao and Guo, Rui},
  title = {Stepwise Diagnostic Evaluation of Chinese Large Language Models: Comparative Study of Common and Rare Diseases},
  journal = {Journal of Medical Internet Research},
  year = {2026},
  volume = {28},
  pages = {e89963--e89963},
  doi = {10.2196/89963}
}
Kostenlos starten →