EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Evaluation of Chatbot Responses to Text-Based Multiple-Choice Questions in Prosthodontic and Restorative Dentistry

Reinhard Chun Wang Chau iD, Khaing Myat Thu iD, Ollie Yiru Yu iD, Richard Tai-Chiu Hsung iD, Denny Chon Pei Wang iD, Manuel Wing Ho Man iD, John Junwen Wang, Walter Yu Hang Lam iD

Dentistry Journal · 21.06.2025 · 13 (7) · 279

DOI: 10.3390/dj13070279 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Dentistry Journal, online erschienen am 21.06.2025. Verfasst von Reinhard Chun Wang Chau und 7 weiteren Autor:innen. Beteiligte Einrichtung: Faculty of Dentistry, The University of Hong Kong, Hong Kong 999077, China. Die Arbeit wurde bisher 43-mal zitiert (Crossref) und nennt 41 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background/Objectives: This study aims to evaluate the response accuracy and quality of three AI chatbots—GPT-4.0, Claude-2, and Llama-2—in answering multiple-choice questions in prosthodontic and restorative dentistry.

Methods

A total of 191 text-based multiple-choice questions were selected from the prosthodontic and restorative dentistry sections of the United States Integrated National Board Dental Examination (INBDE) (n = 80) and the United Kingdom Overseas Registration Examination (ORE) (n = 111). These questions were inputted into the chatbots, and the AI-generated answers were compared with the official answer keys to determine their accuracy. Additionally, two dental specialists independently evaluated the rationales accompanying each chatbot response for accuracy, relevance, and comprehensiveness, categorizing them into four distinct ratings. Chi-square and post hoc Z-tests with Bonferroni adjustment were used to analyze the responses. The inter-rater reliability for evaluating the quality of the rationale ratings among specialists was assessed using Cohen’s Kappa (κ).

Results

GPT-4.0 (65.4%; n = 125/191) demonstrated a significantly higher proportion of correctly answered multiple-choice questions when compared to Claude-2 (41.9%; n = 80/191) (p < 0.017) and Llama-2 (26.2%; n = 50/191) (p < 0.017). Significant differences were observed in the answer accuracy among all of the chatbots (p < 0.001). In terms of the rationale quality, GPT-4.0 (58.1%; n = 111/191) had a significantly higher proportion of “Correct Answer, Correct Rationale” than Claude-2 (37.2%; n = 71/191) (p < 0.017) and Llama-2 (24.1%; n = 46/191) (p < 0.017). Significant differences were observed in the rationale quality among all of the chatbots (p < 0.001). The inter-rater reliability was very high (κ = 0.83).

Conclusions

GPT-4.0 demonstrated the highest accuracy and quality of reasoning in responding to prosthodontic and restorative dentistry questions. This underscores the varying efficacy of AI chatbots within specialized dental contexts.

Schlagwörter

Stichwörter der Autor:innen

artificial intelligencedeep learningmachine learningmouth rehabilitationnatural language processing

Autor:innen und Institutionen

  • Faculty of Dentistry, The University of Hong Kong, Hong Kong 999077, China.
  • Department of Computer Science, Hong Kong Chu Hai College, Hong Kong 999077, China.
  • Musketeers Foundation Institute of Data Science, The University of Hong Kong, Hong Kong 999077, China.

ORCID-Profile: Reinhard Chun Wang Chau, Khaing Myat Thu, Ollie Yiru Yu, Richard Tai-Chiu Hsung, Denny Chon Pei Wang, Manuel Wing Ho Man, Walter Yu Hang Lam

Zitieren

APA

Chau, R. C. W., Thu, K. M., Yu, O. Y., Hsung, R. T., Wang, D. C. P., Man, M. W. H., Wang, J. J., & Lam, W. Y. H. (2025). Evaluation of Chatbot Responses to Text-Based Multiple-Choice Questions in Prosthodontic and Restorative Dentistry. Dentistry Journal, 13(7), 279. https://doi.org/10.3390/dj13070279

Vancouver

Chau RCW, Thu KM, Yu OY, Hsung RT, Wang DCP, Man MWH, et al. Evaluation of Chatbot Responses to Text-Based Multiple-Choice Questions in Prosthodontic and Restorative Dentistry. Dentistry Journal. 2025;13(7):279. doi:10.3390/dj13070279

BibTeX
@article{Chau2025,
  author = {Chau, Reinhard Chun Wang and Thu, Khaing Myat and Yu, Ollie Yiru and Hsung, Richard Tai-Chiu and Wang, Denny Chon Pei and Man, Manuel Wing Ho and Wang, John Junwen and Lam, Walter Yu Hang},
  title = {Evaluation of Chatbot Responses to Text-Based Multiple-Choice Questions in Prosthodontic and Restorative Dentistry},
  journal = {Dentistry Journal},
  year = {2025},
  volume = {13},
  number = {7},
  pages = {279},
  doi = {10.3390/dj13070279}
}
Kostenlos starten →