EUVIMEDEUVIMED entdecken
VergleichsstudieLeitlinieOpen Access · CC BY 4.0

Evaluating large language models using the Type 2 Diabetes Health Education guideline: a comparative analysis of ChatGPT-4.1, Claude-4.0, DeepSeek-V3, and ERNIE Bot 4.5 Turbo

Zhaoxia Huang, Yuxin Bai, Junyue Luo, Zihang Zhang, Chunlan Hu, Xinyu Hu, Liying Ruan, Jiaxin Zeng, Qianhui Jia, Luyao Deng, Meifeng Xiong, Ying Zhou und 3 weitere

Frontiers in Public Health · 2026

DOI: 10.3389/fpubh.2026.1900281 ↗

Kurz gefasst

Vergleichsstudie in der Zeitschrift Frontiers in Public Health, 2026. Verfasst von Zhaoxia Huang und 14 weiteren Autor:innen. Beteiligte Einrichtung: Department of Ophthalmology, The Affiliated Hospital of Southwest Medical University, Luzhou, China. Hauptthemen laut MEDLINE: Large Language Models, Diabetes Mellitus, Type 2, Patient Education as Topic, Health Education. Die Arbeit wurde bisher 1-mal zitiert (Crossref) und nennt 40 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Objective

To systematically evaluate the quality and readability of health information generated by four large language models (LLMs) in response to inquiries regarding type 2 diabetes mellitus (T2DM), using an authoritative Chinese clinical guideline as the reference standard.

Methods

A total of 124 standardized questions were extracted from the Chinese Type 2 Diabetes Popular Science Guidelines. Six endocrinologists and diabetes specialists conducted independent, blind evaluations using the CLEAR tool (Completeness, Lack of false Information, Evidence, Appropriateness, Relevance) and PEMAT-P (Patient Education Materials Assessment Tool for Printable materials). Response characteristics were also recorded. Between-model differences were tested using the Kruskal–Wallis H test with Bonferroni pairwise comparisons.

Results

All four models achieved total CLEAR scores within the “very good” range (19–25), with no significant differences seen between models (χ 2 = 1.985, p = 0.576). No significant differences were observed in the dimensions of Lack of false information (χ 2 = 7.644, p = 0.054), Evidence (χ 2 = 2.309, p = 0.511), and Relevance (χ 2 = 7.516, p = 0.057). However, significant differences emerged in Completeness (χ 2 = 47.661, p < 0.001) and Appropriateness (χ 2 = 88.360, p < 0.001). Claude-4.0 received the lowest score in Completeness (median 4.00, IQR 3.00–5.00) but achieved the highest ranking in Appropriateness (median 4.00, IQR 4.00–5.00). On the PEMAT-P, understandability differed significantly across models (χ 2 = 159.120, p < 0.001), yet all models surpassed the 70% threshold, with ChatGPT-4.1 highest (median 91.91%, IQR 91.91–100.00%). However, despite significant differences among the various models (χ 2 = 354.023, p < 0.001), only ERNIE Bot 4.5 Turbo (median 75.00%, IQR75.00–75.00%) surpassed the 70% threshold, with no single model demonstrating consistent superiority across all dimensions.

Conclusion

Although the four LLMs generally provide accurate and pertinent information regarding type 2 diabetes, enduring limits in actionability and inconsistencies among models in content completeness and understandability restrict their effective use in diabetic patient education. Future development should prioritize stronger step-by-step behavioral guidance and differentiated, scenario-specific model deployment to enhance their value in patient-facing diabetes self-management support.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Large Language ModelsDiabetes Mellitus, Type 2Patient Education as TopicHealth EducationHumansComprehensionChina

Stichwörter der Autor:innen

health information qualitylarge language modelspatient educationreadabilitytype 2 diabetes mellitus

Autor:innen und Institutionen

  • Department of Ophthalmology, The Affiliated Hospital of Southwest Medical University, Luzhou, China.
  • School of Nursing, Southwest Medical University, Luzhou, China.
  • College of Literature, Yancheng Teachers University, Yancheng, China.

Zitieren

APA

Huang, Z., Bai, Y., Luo, J., Zhang, Z., Hu, C., Hu, X., Ruan, L., Zeng, J., Jia, Q., Deng, L., Xiong, M., Zhou, Y., Qi, Z., Lyu, W., & Li, X. (2026). Evaluating large language models using the Type 2 Diabetes Health Education guideline: a comparative analysis of ChatGPT-4.1, Claude-4.0, DeepSeek-V3, and ERNIE Bot 4.5 Turbo. Frontiers in Public Health. https://doi.org/10.3389/fpubh.2026.1900281

Vancouver

Huang Z, Bai Y, Luo J, Zhang Z, Hu C, Hu X, et al. Evaluating large language models using the Type 2 Diabetes Health Education guideline: a comparative analysis of ChatGPT-4.1, Claude-4.0, DeepSeek-V3, and ERNIE Bot 4.5 Turbo. Frontiers in Public Health. 2026. doi:10.3389/fpubh.2026.1900281

BibTeX
@article{Huang2026,
  author = {Huang, Zhaoxia and Bai, Yuxin and Luo, Junyue and Zhang, Zihang and Hu, Chunlan and Hu, Xinyu and Ruan, Liying and Zeng, Jiaxin and Jia, Qianhui and Deng, Luyao and Xiong, Meifeng and Zhou, Ying and Qi, Ziyi and Lyu, Wenxi and Li, Xu},
  title = {Evaluating large language models using the Type 2 Diabetes Health Education guideline: a comparative analysis of ChatGPT-4.1, Claude-4.0, DeepSeek-V3, and ERNIE Bot 4.5 Turbo},
  journal = {Frontiers in Public Health},
  year = {2026},
  doi = {10.3389/fpubh.2026.1900281}
}
Kostenlos starten →