Evaluation of Chatbot Responses to Text-Based Multiple-Choice Questions in Prosthodontic and Restorative Dentistry
Dentistry Journal · 21.06.2025 · 13 (7) · 279
DOI: 10.3390/dj13070279 ↗Kurz gefasst
Fachartikel in der Zeitschrift Dentistry Journal, online erschienen am 21.06.2025. Verfasst von Reinhard Chun Wang Chau und 7 weiteren Autor:innen. Beteiligte Einrichtung: Faculty of Dentistry, The University of Hong Kong, Hong Kong 999077, China. Die Arbeit wurde bisher 43-mal zitiert (Crossref) und nennt 41 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
Background/Objectives: This study aims to evaluate the response accuracy and quality of three AI chatbots—GPT-4.0, Claude-2, and Llama-2—in answering multiple-choice questions in prosthodontic and restorative dentistry.
Methods
A total of 191 text-based multiple-choice questions were selected from the prosthodontic and restorative dentistry sections of the United States Integrated National Board Dental Examination (INBDE) (n = 80) and the United Kingdom Overseas Registration Examination (ORE) (n = 111). These questions were inputted into the chatbots, and the AI-generated answers were compared with the official answer keys to determine their accuracy. Additionally, two dental specialists independently evaluated the rationales accompanying each chatbot response for accuracy, relevance, and comprehensiveness, categorizing them into four distinct ratings. Chi-square and post hoc Z-tests with Bonferroni adjustment were used to analyze the responses. The inter-rater reliability for evaluating the quality of the rationale ratings among specialists was assessed using Cohen’s Kappa (κ).
Results
GPT-4.0 (65.4%; n = 125/191) demonstrated a significantly higher proportion of correctly answered multiple-choice questions when compared to Claude-2 (41.9%; n = 80/191) (p < 0.017) and Llama-2 (26.2%; n = 50/191) (p < 0.017). Significant differences were observed in the answer accuracy among all of the chatbots (p < 0.001). In terms of the rationale quality, GPT-4.0 (58.1%; n = 111/191) had a significantly higher proportion of “Correct Answer, Correct Rationale” than Claude-2 (37.2%; n = 71/191) (p < 0.017) and Llama-2 (24.1%; n = 46/191) (p < 0.017). Significant differences were observed in the rationale quality among all of the chatbots (p < 0.001). The inter-rater reliability was very high (κ = 0.83).
Conclusions
GPT-4.0 demonstrated the highest accuracy and quality of reasoning in responding to prosthodontic and restorative dentistry questions. This underscores the varying efficacy of AI chatbots within specialized dental contexts.
Schlagwörter
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Faculty of Dentistry, The University of Hong Kong, Hong Kong 999077, China.
- Department of Computer Science, Hong Kong Chu Hai College, Hong Kong 999077, China.
- Musketeers Foundation Institute of Data Science, The University of Hong Kong, Hong Kong 999077, China.
ORCID-Profile: Reinhard Chun Wang Chau, Khaing Myat Thu, Ollie Yiru Yu, Richard Tai-Chiu Hsung, Denny Chon Pei Wang, Manuel Wing Ho Man, Walter Yu Hang Lam
Zitieren
APA
Chau, R. C. W., Thu, K. M., Yu, O. Y., Hsung, R. T., Wang, D. C. P., Man, M. W. H., Wang, J. J., & Lam, W. Y. H. (2025). Evaluation of Chatbot Responses to Text-Based Multiple-Choice Questions in Prosthodontic and Restorative Dentistry. Dentistry Journal, 13(7), 279. https://doi.org/10.3390/dj13070279
Vancouver
Chau RCW, Thu KM, Yu OY, Hsung RT, Wang DCP, Man MWH, et al. Evaluation of Chatbot Responses to Text-Based Multiple-Choice Questions in Prosthodontic and Restorative Dentistry. Dentistry Journal. 2025;13(7):279. doi:10.3390/dj13070279
BibTeX
@article{Chau2025,
author = {Chau, Reinhard Chun Wang and Thu, Khaing Myat and Yu, Ollie Yiru and Hsung, Richard Tai-Chiu and Wang, Denny Chon Pei and Man, Manuel Wing Ho and Wang, John Junwen and Lam, Walter Yu Hang},
title = {Evaluation of Chatbot Responses to Text-Based Multiple-Choice Questions in Prosthodontic and Restorative Dentistry},
journal = {Dentistry Journal},
year = {2025},
volume = {13},
number = {7},
pages = {279},
doi = {10.3390/dj13070279}
}
EUVIMED