Evaluation of Arabic-Language AI Chatbot Responses to Migraine-Related Questions: A Comparative Cross-Sectional Study
Journal of Clinical Medicine · 19.05.2026 · 15 (10) · 3908
DOI: 10.3390/jcm15103908 ↗Kurz gefasst
Querschnittstudie in der Zeitschrift Journal of Clinical Medicine, online erschienen am 19.05.2026. Verfasst von Danah Aljaafari und 8 weiteren Autor:innen. Beteiligte Einrichtung: Department of Neurology, College of Medicine, Imam Abdulrahman bin Faisal University, Dammam 34212, Saudi Arabia. Die Arbeit wurde bisher 1-mal zitiert (Crossref) und nennt 43 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
Background/Objectives: Migraine is a common and disabling neurological disorder, and many individuals increasingly seek information online. With the growing use of large language models (LLMs), such as ChatGPT, for patient education, concerns have emerged regarding the quality and reliability of the responses they generate, particularly in Arabic, where evidence remains limited. This study aimed to evaluate the reliability, quality, and accuracy of Arabic-language responses to frequently asked questions (FAQs) about migraine.
Methods
A total of 25 FAQs were selected using a multisource approach and entered into four LLMs (ChatGPT-4.1, Gemini 3 Flash, DeepSeek-V3.2, and Grok 4.1), generating 100 responses. Responses were evaluated by a panel of expert neurologists using the modified DISCERN (mDISCERN), Global Quality Scale (GQS), and an accuracy scale. Inter-rater reliability was assessed using the intraclass correlation coefficient (ICC).
Results
Significant differences were observed between chatbots for mDISCERN and GQS (both p < 0.001), whereas accuracy did not differ significantly across models (p = 0.072). DeepSeek and Grok demonstrated the highest mDISCERN scores (34.07 ± 1.31 and 34.29 ± 2.59, respectively), while DeepSeek achieved the highest GQS (4.95 ± 0.13). The clearest between-model differences were observed in source transparency and communication of uncertainty. Inter-rater reliability was good across all instruments (ICC range, 0.799–0.831).
Conclusions
Medical content generated by the chatbots was broadly comparable, whereas important differences were observed in how that content was communicated. These tools may support patient education; however, their use should remain guided by clinical oversight and professional judgment.
Schlagwörter
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Department of Neurology, College of Medicine, Imam Abdulrahman bin Faisal University, Dammam 34212, Saudi Arabia.
- College of Medicine, King Faisal University, Al-Ahsa 36362, Saudi Arabia.
- Internal Medicine Department, Hail University, Hail 55476, Saudi Arabia.
ORCID-Profile: Hussain Khalifa Aljumah, Majed M Alabdali, Walid Alesefir
Zitieren
APA
Aljaafari, D., Aljumah, H. K., Alzuwayr, M. A., Al-Essa, Y. M., Alradhi, H. A., Alabdali, M. M., Almuslim, N., Alqarni, M. A., & Alesefir, W. (2026). Evaluation of Arabic-Language AI Chatbot Responses to Migraine-Related Questions: A Comparative Cross-Sectional Study. Journal of Clinical Medicine, 15(10), 3908. https://doi.org/10.3390/jcm15103908
Vancouver
Aljaafari D, Aljumah HK, Alzuwayr MA, Al-Essa YM, Alradhi HA, Alabdali MM, et al. Evaluation of Arabic-Language AI Chatbot Responses to Migraine-Related Questions: A Comparative Cross-Sectional Study. Journal of Clinical Medicine. 2026;15(10):3908. doi:10.3390/jcm15103908
BibTeX
@article{Aljaafari2026,
author = {Aljaafari, Danah and Aljumah, Hussain Khalifa and Alzuwayr, Mujtaba Abbas and Al-Essa, Yaqeen Mohammed and Alradhi, Hassan Ali and Alabdali, Majed M and Almuslim, Nora and Alqarni, Mustafa Ahmed and Alesefir, Walid},
title = {Evaluation of Arabic-Language AI Chatbot Responses to Migraine-Related Questions: A Comparative Cross-Sectional Study},
journal = {Journal of Clinical Medicine},
year = {2026},
volume = {15},
number = {10},
pages = {3908},
doi = {10.3390/jcm15103908}
}
EUVIMED