Safety, accuracy, empathy, reliability, and readability of large language model responses to patient health questions about adenomyosis: A comparative study
Sicherheit, Genauigkeit, Empathie, Zuverlässigkeit und Lesbarkeit von Large-Language-Model-Antworten auf Patient:innenfragen zu Adenomyose: vergleichende Studie
International Journal of Gynecology & Obstetrics · 09.11.2026
DOI: 10.1002/ijgo.71446 ↗Auf einen Blick
160 Antworten von 4 LLM-Chatbots zu 40 Adenomyose-Fragen wurden bewertet. Die Antworten waren insgesamt meist sicher, die Qualität variierte aber je nach Modell und Bewertungsbereich.
Fragestellung
Wie unterscheiden sich Sicherheit, Genauigkeit, Empathie, Zuverlässigkeit, Informationsqualität und Lesbarkeit von LLM-Antworten zu Patient:innenfragen über Adenomyose?
Methode
Vergleichende Querschnittsauswertung nach CHART; Fragen vom 13. Juni bis 15. Juni 2026 in Wuhan, China; 5 Gynäkologie-Expert:innen bewerteten die Antworten der 4 Chatbots.
Ergebnisse
160 Modell-Frage-Antworten wurden beurteilt. Übereinstimmung der Bewerter:innen: Fleiss' kappa 0,892 für Sicherheit, ICC 0,768 bis 0,886 für die übrigen Metriken. Genauigkeit und Empathie unterschieden sich signifikant; ChatGPT hatte die höchste mediane Genauigkeitsbewertung.
Schlussfolgerung der Autor:innen
LLMs lieferten insgesamt sichere und moderat hochwertige Antworten zu Adenomyose-Fragen, die Leistung variierte jedoch zwischen den Modellen und Bewertungsbereichen.
Kernaussagen
- Meist als sicher eingestufte Antworten
- Unterschiede bei Genauigkeit, Empathie und Informationsqualität
- Lesbarkeit je nach Modell unterschiedlich
Deutsche Zusammenfassung: KI-gestützt aus dem Original-Abstract erstellt; alle Zahlen wurden automatisch mit dem Original abgeglichen. Maßgeblich ist der Original-Abstract.
Kurz gefasst
Fachartikel in der Zeitschrift International Journal of Gynecology & Obstetrics, online erschienen am 09.11.2026. Verfasst von Lingwei Zhang und 5 weiteren Autor:innen. Beteiligte Einrichtung: Department of Nursing, Tongji Hospital, Tongji Medical College, Huazhong University of Science and Technology, Wuhan, China. Die Arbeit nennt 34 Literaturangaben.
Abstract (Originalsprache)
Background
Patients increasingly use large language model (LLM)‐based chatbots to seek health information. Adenomyosis is a common gynecological condition associated with heavy menstrual bleeding, dysmenorrhea, pelvic pain, infertility, and pregnancy‐related concerns. The quality and safety of chatbot‐generated patient information is clinically relevant.
Objective
Our objective was to compare the safety, accuracy, empathy, reliability, information quality, and readability of LLM responses to patient health questions about adenomyosis.
Methods
This comparative cross‐sectional evaluation was designed and reported according to the CHART checklist for chatbot health advice studies. Standardized chatbot queries were posed from June 13 to June 15, 2026, in Wuhan, China. A 40‐item adenomyosis patient question set was submitted to four LLM‐based chatbots: ChatGPT, Gemini, DeepSeek, and Doubao. Five trained gynecology experts evaluated the safety, accuracy, empathy, DISCERN score, EQIP (Ensuring Quality Information for Patients) score, Journal of the American Medical Association (JAMA) benchmark criteria, and Global Quality Scale score using prespecified criteria; readability was calculated algorithmically using automated text‐processing procedures. Safety was coded as 0 for safe and 1 for potentially unsafe. Inter‐rater agreement was assessed using Fleiss' kappa for safety and intraclass correlation coefficient (2,1) for the remaining expert‐rated metrics. Between‐model comparisons used paired nonparametric methods, with Kendall's W and 95% confidence intervals reported as the omnibus effect‐size measure for Friedman tests.
Results
A total of 160 model–question responses were evaluated. Inter‐rater agreement was substantial to excellent across metrics, with Fleiss' kappa of 0.892 for safety and intraclass correlation coefficient values ranging from 0.768 to 0.886 for the remaining metrics. Most responses were classified as safe; observed safe‐response proportions differed descriptively across models, and no pairwise safety comparison remained statistically significant after Benjamini–Hochberg adjustment. Accuracy and empathy differed significantly among models (both P < 0.001): ChatGPT achieved the highest median accuracy score. DISCERN, EQIP, and JAMA scores differed significantly among models, whereas Global Quality Scale scores did not. All readability indices differed significantly among models: Doubao had the highest grade‐level readability scores and the lowest Flesch Reading Ease score, whereas Gemini had comparatively easier readability.
Conclusions
LLMs produced generally safe and moderately high‐quality responses to adenomyosis‐related patient questions, but performance varied by model and by evaluation domain. These findings support the need for transparent, multidimensional assessment of LLM‐generated health advice before patient‐facing use.
Schlagwörter
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Department of Nursing, Tongji Hospital, Tongji Medical College, Huazhong University of Science and Technology, Wuhan, China.
- Department of Spine Surgery, Third Affiliated Hospital of Soochow University, Changzhou, China.
ORCID-Profile: Lingwei Zhang
Zitieren
APA
Zhang, L., Wu, F., Ye, D., Wu, Z., Ma, H., & Liu, L. (2026). Safety, accuracy, empathy, reliability, and readability of large language model responses to patient health questions about adenomyosis: A comparative study. International Journal of Gynecology & Obstetrics. https://doi.org/10.1002/ijgo.71446
Vancouver
Zhang L, Wu F, Ye D, Wu Z, Ma H, Liu L. Safety, accuracy, empathy, reliability, and readability of large language model responses to patient health questions about adenomyosis: A comparative study. International Journal of Gynecology & Obstetrics. 2026. doi:10.1002/ijgo.71446
BibTeX
@article{Zhang2026,
author = {Zhang, Lingwei and Wu, Fen and Ye, Duanlian and Wu, Zhishui and Ma, Huan and Liu, Li},
title = {Safety, accuracy, empathy, reliability, and readability of large language model responses to patient health questions about adenomyosis: A comparative study},
journal = {International Journal of Gynecology & Obstetrics},
year = {2026},
doi = {10.1002/ijgo.71446}
}
EUVIMED