Reliability, Readability, and Safety of Generative AI Chatbots for Osteoporosis Health Counseling: A Multidimensional Cross‐Sectional Comparative Study
Health Science Reports · 2026-10 · 9 (10)
DOI: 10.1002/hsr2.73381 ↗Kurz gefasst
Fachartikel in der Zeitschrift Health Science Reports, online erschienen am 10.10.2026. Verfasst von Huimin Xu und 5 weiteren Autor:innen. Beteiligte Einrichtung: Yueyang Hospital of Integrated Traditional Chinese and Western Medicine, Shanghai University of Traditional Chinese Medicine Shanghai China. Die Arbeit nennt 34 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-NC-ND 4.0).
Abstract (Originalsprache)
Background and Aims Generative artificial intelligence chatbots are increasingly used for health information, but their reliability, readability, and safety in osteoporosis counseling remain uncertain. This study compared four widely available chatbots under standardized conditions.
Methods
Twenty‐five standardized osteoporosis questions were submitted once to ChatGPT‐5, Gemini 2.5 Pro, Copilot, and Perplexity Pro, generating 100 responses. Two model‐blinded specialists evaluated the final responses using DISCERN, EQIP, the Global Quality Scale, JAMA benchmarks, and the NOHARM framework. Six automated indices assessed readability. Because each question was answered by all four models, outcomes were compared using Friedman tests, followed by paired Wilcoxon signed‐rank tests with Holm adjustment.
Results
DISCERN scores differed across models ( p < 0.001; Kendall's W = 0.82, 95% CI 0.76–0.89). Median scores were 55.0 for Perplexity Pro, 54.0 for Copilot, 44.0 for Gemini 2.5 Pro, and 42.0 for ChatGPT‐5. JAMA benchmark scores did not differ significantly ( p = 0.63). All six readability indices differed across models (all p < 0.001). Perplexity Pro had the highest median Flesch–Kincaid Grade Level score at 16.18, whereas Copilot had the lowest at 9.49; all models exceeded the sixth‐grade benchmark. The qualitative safety assessment also identified clinically relevant omissions and an unsupported claim.
Conclusion
In this set of osteoporosis questions, retrieval‐enabled systems achieved higher scores on selected information‐quality measures but produced more difficult text. These findings are limited to the tested models, prompts, and evaluation period and do not establish a universal effect of retrieval augmentation.
Schlagwörter
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Yueyang Hospital of Integrated Traditional Chinese and Western Medicine, Shanghai University of Traditional Chinese Medicine Shanghai China.
ORCID-Profile: Huimin Xu, Liwen Ding
Zitieren
APA
Xu, H., Guo, L., Shen, L., Zhu, T., Huang, X., & Ding, L. (2026). Reliability, Readability, and Safety of Generative AI Chatbots for Osteoporosis Health Counseling: A Multidimensional Cross‐Sectional Comparative Study. Health Science Reports, 9(10). https://doi.org/10.1002/hsr2.73381
Vancouver
Xu H, Guo L, Shen L, Zhu T, Huang X, Ding L. Reliability, Readability, and Safety of Generative AI Chatbots for Osteoporosis Health Counseling: A Multidimensional Cross‐Sectional Comparative Study. Health Science Reports. 2026;9(10). doi:10.1002/hsr2.73381
BibTeX
@article{Xu2026,
author = {Xu, Huimin and Guo, Liang and Shen, Li and Zhu, Ting and Huang, Xiaona and Ding, Liwen},
title = {Reliability, Readability, and Safety of Generative AI Chatbots for Osteoporosis Health Counseling: A Multidimensional Cross‐Sectional Comparative Study},
journal = {Health Science Reports},
year = {2026},
volume = {9},
number = {10},
doi = {10.1002/hsr2.73381}
}
EUVIMED