Evaluating large language models using the Type 2 Diabetes Health Education guideline: a comparative analysis of ChatGPT-4.1, Claude-4.0, DeepSeek-V3, and ERNIE Bot 4.5 Turbo
Frontiers in Public Health · 2026
DOI: 10.3389/fpubh.2026.1900281 ↗Kurz gefasst
Vergleichsstudie in der Zeitschrift Frontiers in Public Health, 2026. Verfasst von Zhaoxia Huang und 14 weiteren Autor:innen. Beteiligte Einrichtung: Department of Ophthalmology, The Affiliated Hospital of Southwest Medical University, Luzhou, China. Hauptthemen laut MEDLINE: Large Language Models, Diabetes Mellitus, Type 2, Patient Education as Topic, Health Education. Die Arbeit wurde bisher 1-mal zitiert (Crossref) und nennt 40 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Themen: Diabetes mellitus, Künstliche Intelligenz in der Pflege
Abstract (Originalsprache)
Objective
To systematically evaluate the quality and readability of health information generated by four large language models (LLMs) in response to inquiries regarding type 2 diabetes mellitus (T2DM), using an authoritative Chinese clinical guideline as the reference standard.
Methods
A total of 124 standardized questions were extracted from the Chinese Type 2 Diabetes Popular Science Guidelines. Six endocrinologists and diabetes specialists conducted independent, blind evaluations using the CLEAR tool (Completeness, Lack of false Information, Evidence, Appropriateness, Relevance) and PEMAT-P (Patient Education Materials Assessment Tool for Printable materials). Response characteristics were also recorded. Between-model differences were tested using the Kruskal–Wallis H test with Bonferroni pairwise comparisons.
Results
All four models achieved total CLEAR scores within the “very good” range (19–25), with no significant differences seen between models (χ 2 = 1.985, p = 0.576). No significant differences were observed in the dimensions of Lack of false information (χ 2 = 7.644, p = 0.054), Evidence (χ 2 = 2.309, p = 0.511), and Relevance (χ 2 = 7.516, p = 0.057). However, significant differences emerged in Completeness (χ 2 = 47.661, p < 0.001) and Appropriateness (χ 2 = 88.360, p < 0.001). Claude-4.0 received the lowest score in Completeness (median 4.00, IQR 3.00–5.00) but achieved the highest ranking in Appropriateness (median 4.00, IQR 4.00–5.00). On the PEMAT-P, understandability differed significantly across models (χ 2 = 159.120, p < 0.001), yet all models surpassed the 70% threshold, with ChatGPT-4.1 highest (median 91.91%, IQR 91.91–100.00%). However, despite significant differences among the various models (χ 2 = 354.023, p < 0.001), only ERNIE Bot 4.5 Turbo (median 75.00%, IQR75.00–75.00%) surpassed the 70% threshold, with no single model demonstrating consistent superiority across all dimensions.
Conclusion
Although the four LLMs generally provide accurate and pertinent information regarding type 2 diabetes, enduring limits in actionability and inconsistencies among models in content completeness and understandability restrict their effective use in diabetic patient education. Future development should prioritize stronger step-by-step behavioral guidance and differentiated, scenario-specific model deployment to enhance their value in patient-facing diabetes self-management support.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Department of Ophthalmology, The Affiliated Hospital of Southwest Medical University, Luzhou, China.
- School of Nursing, Southwest Medical University, Luzhou, China.
- College of Literature, Yancheng Teachers University, Yancheng, China.
Zitieren
APA
Huang, Z., Bai, Y., Luo, J., Zhang, Z., Hu, C., Hu, X., Ruan, L., Zeng, J., Jia, Q., Deng, L., Xiong, M., Zhou, Y., Qi, Z., Lyu, W., & Li, X. (2026). Evaluating large language models using the Type 2 Diabetes Health Education guideline: a comparative analysis of ChatGPT-4.1, Claude-4.0, DeepSeek-V3, and ERNIE Bot 4.5 Turbo. Frontiers in Public Health. https://doi.org/10.3389/fpubh.2026.1900281
Vancouver
Huang Z, Bai Y, Luo J, Zhang Z, Hu C, Hu X, et al. Evaluating large language models using the Type 2 Diabetes Health Education guideline: a comparative analysis of ChatGPT-4.1, Claude-4.0, DeepSeek-V3, and ERNIE Bot 4.5 Turbo. Frontiers in Public Health. 2026. doi:10.3389/fpubh.2026.1900281
BibTeX
@article{Huang2026,
author = {Huang, Zhaoxia and Bai, Yuxin and Luo, Junyue and Zhang, Zihang and Hu, Chunlan and Hu, Xinyu and Ruan, Liying and Zeng, Jiaxin and Jia, Qianhui and Deng, Luyao and Xiong, Meifeng and Zhou, Ying and Qi, Ziyi and Lyu, Wenxi and Li, Xu},
title = {Evaluating large language models using the Type 2 Diabetes Health Education guideline: a comparative analysis of ChatGPT-4.1, Claude-4.0, DeepSeek-V3, and ERNIE Bot 4.5 Turbo},
journal = {Frontiers in Public Health},
year = {2026},
doi = {10.3389/fpubh.2026.1900281}
}
EUVIMED