Evaluation of validity, reliability, and readability of AI chatbots for gestational diabetes mellitus: a multi-model comparative study
Bewertung von Validität, Reliabilität und Lesbarkeit von KI-Chatbots bei Gestationsdiabetes mellitus: eine multi-modell Vergleichsstudie
Frontiers in Public Health · 04.02.2026 · 14
DOI: 10.3389/fpubh.2026.1760871 ↗Auf einen Blick
Sechs KI-Chatbots wurden zu Gestationsdiabetes mellitus geprüft. ChatGPT-5 erzielte die höchste Genauigkeit; die Modelle waren jedoch schwer lesbar und nur begrenzt transparent.
Fragestellung
Wie valide, reliabel und gut lesbar sind KI-Chatbots zu Gestationsdiabetes mellitus?
Methode
Vergleichsstudie mit 6 Chatbots. 200 MCQs zu GDM aus drei Datenquellen; jede Frage dreimal gestellt. Für Public-Info: 15 Fragen, Bewertung durch 2 Gynäkolog:innen mit DISCERN, EQIP, GQS und JAMA; Lesbarkeit mit mehreren Indizes.
Ergebnisse
Die MCQ-Genauigkeit unterschied sich signifikant (p < 0,0001). ChatGPT-5 hatte die höchste mittlere Genauigkeit (92,17 %). Bei Public-Info hatte ChatGPT-5 die besten Reliabilitätswerte; JAMA war bei allen Modellen niedrig (0–0,07/4).
Einschränkungen
JAMA-Scores waren bei allen Modellen niedrig; alle Texte lagen über dem empfohlenen Leselevel.
Schlussfolgerung der Autor:innen
Moderne KI-Chatbots können zu GDM allgemein genaue und moderat zuverlässige Informationen liefern, sind aber wegen begrenzter Transparenz und hoher Lesbarkeit noch nicht als alleinige Patientenressource geeignet.
Kernaussagen
- ChatGPT-5 erreichte die höchste MCQ-Genauigkeit (92,17 %)
- Alle Modelle lagen über dem empfohlenen sechsten Leselevel
- JAMA-Scores waren bei allen Modellen sehr niedrig (0–0,07/4)
Deutsche Zusammenfassung: KI-gestützt aus dem Original-Abstract erstellt; alle Zahlen wurden automatisch mit dem Original abgeglichen. Maßgeblich ist der Original-Abstract.
Kurz gefasst
Vergleichsstudie in der Zeitschrift Frontiers in Public Health, online erschienen am 04.02.2026. Verfasst von Xinxin Wang und 5 weiteren Autor:innen. Beteiligte Einrichtung: Department of Obstetrics, Shenzhen Nanshan Maternity and Child Healthcare Hospital, Shenzhen, China. Hauptthemen laut MEDLINE: Diabetes, Gestational, Artificial Intelligence, Comprehension. Die Arbeit wurde bisher 9-mal zitiert (Crossref) und nennt 30 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Themen: Diabetes mellitus
Abstract (Originalsprache)
Background
Gestational diabetes mellitus (GDM) is increasingly prevalent worldwide and is associated with substantial short- and long-term risks for mothers and offspring, making high-quality, accessible health information essential. At the same time, artificial intelligence (AI) chatbots based on large language models are being widely used for health queries, yet their accuracy, reliability and readability in the context of GDM remain unclear.
Methods
We first evaluated six AI chatbots (ChatGPT-5, ChatGPT-4o, DeepSeek-V3.2, DeepSeek-R1, Gemini 2.5 Pro and Claude Sonnet 4.5) using 200 single-best-answer multiple-choice questions (MCQs) on GDM drawn from MedQA, MedMCQA and the Chinese National Medical Examination item bank, covering four domains: epidemiology and risk factors, clinical manifestations and diagnosis, maternal and neonatal outcomes, and management and treatment. Each item was posed three times to every model under a standardized prompting protocol, and accuracy was defined as the proportion of correctly answered questions. For public-facing information, we identified 15 core GDM education questions using Google Trends and expert review, and queried four chatbots (ChatGPT-5, DeepSeek-V3.2, Claude Sonnet 4.5 and Gemini 2.5 Pro). Two obstetricians independently assessed reliability using DISCERN, EQIP, GQS and JAMA benchmarks, and readability was quantified using ARI, CL, FKGL, FRES, GFI and SMOG indices.
Results
Overall MCQ accuracy differed significantly across the six chatbots ( p < 0.0001), with ChatGPT-5 achieving the highest mean accuracy (92.17%) and DeepSeek-V3.2 and Gemini 2.5 Pro performing comparably well, while ChatGPT-4o, DeepSeek-R1 and Claude Sonnet 4.5 scored lower. Newer model generations (ChatGPT-5 vs. ChatGPT-4o; DeepSeek-V3.2 vs. DeepSeek-R1) consistently outperformed their predecessors across all four domains. Among the four models evaluated on public-education questions, ChatGPT-5 achieved the highest reliability scores (DISCERN 42.53 ± 7.20; EQIP 71.67 ± 6.17), whereas Claude Sonnet 4.5, DeepSeek-V3.2 and Gemini 2.5 Pro scored lower. JAMA scores were uniformly low (0–0.07/4), reflecting poor transparency. All models produced text above the recommended sixth-grade reading level; ChatGPT-5 showed the most favorable readability profile (for example, FKGL 7.43 ± 2.42, FRES 62.47 ± 13.51) but still did not meet guideline targets.
Conclusion
Contemporary AI chatbots can generate generally accurate and moderately reliable GDM-related information, with newer model generations showing clear gains in diagnostic validity. However, limited transparency and systematically high reading levels indicate that these tools are not yet suitable as stand-alone resources for GDM patient education and should be used as adjuncts to clinician counseling and professionally curated materials.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Department of Obstetrics, Shenzhen Nanshan Maternity and Child Healthcare Hospital, Shenzhen, China.
- Department of Urology, South China Hospital, Medical School, Shenzhen University, Shenzhen, China.
- Department of Urology, Lanzhou University Second Hospital, Lanzhou University, Lanzhou, China.
Zitieren
APA
Wang, X., Lin, S., Liu, H., Li, C., Zhou, L., & Li, R. (2026). Evaluation of validity, reliability, and readability of AI chatbots for gestational diabetes mellitus: a multi-model comparative study. Frontiers in Public Health, 14. https://doi.org/10.3389/fpubh.2026.1760871
Vancouver
Wang X, Lin S, Liu H, Li C, Zhou L, Li R. Evaluation of validity, reliability, and readability of AI chatbots for gestational diabetes mellitus: a multi-model comparative study. Frontiers in Public Health. 2026;14. doi:10.3389/fpubh.2026.1760871
BibTeX
@article{Wang2026,
author = {Wang, Xinxin and Lin, Shuyan and Liu, Hui and Li, Chuanqing and Zhou, Li and Li, Rongkang},
title = {Evaluation of validity, reliability, and readability of AI chatbots for gestational diabetes mellitus: a multi-model comparative study},
journal = {Frontiers in Public Health},
year = {2026},
volume = {14},
doi = {10.3389/fpubh.2026.1760871}
}
EUVIMED