Exploring LLM-based chatbot effectiveness in answering questions related to the risks and benefits of orthognathic treatment: a cross-sectional study
European Journal of Orthodontics · 03.04.2026 · 48 (3)
DOI: 10.1093/ejo/cjag034 ↗Kurz gefasst
Querschnittstudie in der Zeitschrift European Journal of Orthodontics, online erschienen am 13.05.2026. Verfasst von Robert Samuel David Smyth und 1 weiteren Autor:innen. Beteiligte Einrichtung: Orthodontic Unit, UCL Eastman Dental Institute, Rockefeller Building, 21 University Street, London WC1E 6DE, United Kingdom. Hauptthemen laut MEDLINE: Large Language Models, Orthognathic Surgical Procedures. Die Arbeit wurde bisher 2-mal zitiert (Crossref) und nennt 24 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
Objective
To assess the accuracy, reliability, quality, and readability of responses generated by three large language model chatbots, ChatGPT4o, Microsoft Copilot, and Google Gemini 2.5 Flash, when answering common patient questions about the risks and benefits of orthognathic treatment. Materials and Methods Twenty frequently searched questions were identified via Google and entered into each chatbot. Responses were evaluated using validated scoring systems for accuracy, modified DISCERN, global quality scale (GQS), and Flesch Reading Ease. Intra- and inter-rater reliability was assessed using Cohen’s kappa and intra-class correlation coefficients. Non-parametric tests were applied due to non-normal data distribution.
Results
Copilot achieved the highest reliability and quality scores, with significant differences observed in modified DISCERN (P < 0.001) and GQS (P = 0.046). Post hoc tests confirmed Copilot significantly outperformed ChatGPT. Accuracy scores did not differ significantly (P = 0.704). Readability varied significantly with Gemini and ChatGPT producing more accessible responses than Copilot. Intra- and inter-rater reliability scores were substantial to excellent for categorical measures and excellent for readability.
Conclusions
Copilot provided the most reliable and high-quality responses, whilst ChatGPT and Gemini offered greater readability ease. Despite these strengths, variability in accuracy and reliability highlights the need for caution. Chatbots should be considered as supplementary tools, and patients should verify information with qualified professionals.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Orthodontic Unit, UCL Eastman Dental Institute, Rockefeller Building, 21 University Street, London WC1E 6DE, United Kingdom.
ORCID-Profile: Robert Samuel David Smyth
Zitieren
APA
Smyth, R. S. D., & Tsichlaki, A. (2026). Exploring LLM-based chatbot effectiveness in answering questions related to the risks and benefits of orthognathic treatment: a cross-sectional study. European Journal of Orthodontics, 48(3). https://doi.org/10.1093/ejo/cjag034
Vancouver
Smyth RSD, Tsichlaki A. Exploring LLM-based chatbot effectiveness in answering questions related to the risks and benefits of orthognathic treatment: a cross-sectional study. European Journal of Orthodontics. 2026;48(3). doi:10.1093/ejo/cjag034
BibTeX
@article{Smyth2026,
author = {Smyth, Robert Samuel David and Tsichlaki, Aliki},
title = {Exploring LLM-based chatbot effectiveness in answering questions related to the risks and benefits of orthognathic treatment: a cross-sectional study},
journal = {European Journal of Orthodontics},
year = {2026},
volume = {48},
number = {3},
doi = {10.1093/ejo/cjag034}
}
EUVIMED