Assessment of accuracy, reliability, readability, and semantic consistency of generative AI responses to rephrased orthodontic patient questions
European Journal of Orthodontics · 2026
DOI: 10.1093/ejo/cjag024 ↗Kurz gefasst
Vergleichsstudie in der Zeitschrift European Journal of Orthodontics, 2026. Verfasst von Panchali Batra und 6 weiteren Autor:innen. Beteiligte Einrichtung: Department of Orthodontics and Dentofacial Orthopaedics, Faculty of Dentistry, Jamia Millia Islamia, New Delhi 110025, India. Hauptthemen laut MEDLINE: Generative Artificial Intelligence, Comprehension, Orthodontics, Patient Education as Topic. Die Arbeit wurde bisher 1-mal zitiert (Crossref) und nennt 35 Literaturangaben.
Abstract (Originalsprache)
Background
Patients increasingly consult generative artificial intelligence (GenAI) chatbots for orthodontic information. However, as a single question may be phrased in multiple ways, the consistency of responses remains uncertain.
Objective
To evaluate and compare the accuracy, quality, reliability, readability, and semantic stability of five GenAI Chabot models in answering common orthodontic patient questions and their rephrased variants. Materials and Methods 13 frequently asked orthodontic questions were identified and rephrased once, yielding 26 prompts. Each prompt was submitted to five AI models under standardized conditions. All models were accessed via publicly available web-based chat interfaces, including ChatGPT-3.5, ChatGPT-4, Bing Copilot, Google Gemini 1.0 Pro, and LLaMA-3-70B-Instruct Five orthodontists independently rated responses using a 4-point accuracy Likert scale, Global Quality Scale, modified DISCERN checklist, and Flesch Reading Ease Score (FRES). Readability was additionally assessed by students in grades 5–8. Semantic stability was determined by comparing ratings for original versus rephrased prompts. Statistical analyses included ANOVA, chi-square tests, and intraclass correlation coefficients (ICC).
Results
ChatGPT-4 ranked second but exhibited moderate consistency (54%). Gemini showed good stability (92%) but lower quality. Bing Copilot and ChatGPT-3.5 underperformed, especially in consistency (46% and 23%). All models produced responses with low readability (Flesch scores 45–49).
Conclusions
GenAI chatbots provide generally accurate orthodontic information, but performance depends on question phrasing. Only LLaMa and Gemini show high consistency. Better consistency, readability, and evidence-based accuracy are needed before the use of LLMs in patient education.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Department of Orthodontics and Dentofacial Orthopaedics, Faculty of Dentistry, Jamia Millia Islamia, New Delhi 110025, India.
- Department of Orthodontics and Dentofacial Orthopaedics ECHS Dental Officer, BRV - Integrated veterans complex, Bengaluru, Karnataka 560001, India.
- Department of Orthodontics and Dentofacial Orthopaedics, Jss dental college and hospital, Mysuru, Karnataka 570015, India.
- Department of Orthodontics and Dentofacial Orthopaedics, King George Medical University, Lucknow, Uttar Pradesh 226003, India.
- Department of Orthodontics and Dentofacial Orthopaedics, Faculty of Dental Sciences, Banaras Hindu University, Uttar Pradesh 221005, India.
- Department of Orthodontics and Dentofacial Orthopaedics, Institute of Dental Studies & Technologies, Modinagar, Ghaziabad, Uttar Pradesh 201204, India.
ORCID-Profile: Panchali Batra, Eshwar Puttaswamy
Zitieren
APA
Batra, P., Puttaswamy, E., Hurkadle, J., Shastri, D., Sharma, V., Raza, M., & Jakhal, M. (2026). Assessment of accuracy, reliability, readability, and semantic consistency of generative AI responses to rephrased orthodontic patient questions. European Journal of Orthodontics. https://doi.org/10.1093/ejo/cjag024
Vancouver
Batra P, Puttaswamy E, Hurkadle J, Shastri D, Sharma V, Raza M, et al. Assessment of accuracy, reliability, readability, and semantic consistency of generative AI responses to rephrased orthodontic patient questions. European Journal of Orthodontics. 2026. doi:10.1093/ejo/cjag024
BibTeX
@article{Batra2026,
author = {Batra, Panchali and Puttaswamy, Eshwar and Hurkadle, JyothiKiran and Shastri, Dipti and Sharma, Vipul and Raza, Monis and Jakhal, Muskan},
title = {Assessment of accuracy, reliability, readability, and semantic consistency of generative AI responses to rephrased orthodontic patient questions},
journal = {European Journal of Orthodontics},
year = {2026},
doi = {10.1093/ejo/cjag024}
}
EUVIMED