EUVIMEDObjevte EUVIMED
Vergleichsstudie

Evaluating AI Chatbot Performance Against Prosthodontic Residents on the National Prosthodontic Resident Examination (NPRE) from 2011 to 2023

Despoina Bompolaki, Esha Mukherjee, Kelsey A K Lee, Christopher Yoon, Hongseok An

The International Journal of Prosthodontics · 2025

DOI: 10.11607/ijp.9486 ↗

At a glance

Vergleichsstudie in The International Journal of Prosthodontics, 2025. Written by Despoina Bompolaki and 4 co-authors. Major MEDLINE topics: Prosthodontics, Internship and Residency, Artificial Intelligence, Educational Measurement.

AbstraktSubjectsBibliografické údajeCiteRelated studies

Abstract (original language)

PURPOSE: To compare the performance of seven artificial intelligence (AI) chatbots (ChatGPT-3.5, ChatGPT-4, ChatGPT-4o, Gemini, Claude 3, Copilot, and Llama) to that of prosthodontic residents in the USA on the National Prosthodontic Resident Examination (NPRE) from 2011 to 2023. MATERIALS AND METHODS: NPRE exam files and answer keys were obtained from the American College of Prosthodontists (ACP). AI-generated responses were collected using standardized prompts that requested only letter answer choices. Each chatbot's first response was recorded and compared to the official exam key. Performance was evaluated as the percentage of correct responses, and results were compared to the average scores of human participants. Statistical analysis included repeated measures ANOVA and Bonferroni post-hoc comparisons. RESULTS: Statistically significant differences were found between the different groups (P < .001). ChatGPT-3.5 achieved the highest scores, followed by Claude 3 and Llama, while Copilot performed significantly worse than residents. Other chatbots demonstrated performance levels comparable to human participants, but with notable variability across AI models. The findings suggest that AI chatbots, particularly ChatGPT-3.5, can perform at or above the level of prosthodontic residents on standardized examinations. However, differences in chatbot capabilities, training data, and inconsistencies in performance raise concerns about reliability. While AI has potential as a supplementary learning tool, its use should be guided by critical evaluation. CONCLUSIONS: AI chatbots demonstrate adequate performance on specialized academic exams, but variability across models highlights the need for further assessment before integration into dental education and professional practice.

Subjects

MeSH terms (MEDLINE)

ProsthodonticsInternship and ResidencyArtificial IntelligenceEducational MeasurementHumansUnited StatesGenerative Artificial Intelligence

Cite

APA

Bompolaki, D., Mukherjee, E., Lee, K. A. K., Yoon, C., & An, H. (2025). Evaluating AI Chatbot Performance Against Prosthodontic Residents on the National Prosthodontic Resident Examination (NPRE) from 2011 to 2023. The International Journal of Prosthodontics. https://doi.org/10.11607/ijp.9486

Vancouver

Bompolaki D, Mukherjee E, Lee KAK, Yoon C, An H. Evaluating AI Chatbot Performance Against Prosthodontic Residents on the National Prosthodontic Resident Examination (NPRE) from 2011 to 2023. The International Journal of Prosthodontics. 2025. doi:10.11607/ijp.9486

BibTeX
@article{Bompolaki2025,
  author = {Bompolaki, Despoina and Mukherjee, Esha and Lee, Kelsey A K and Yoon, Christopher and An, Hongseok},
  title = {Evaluating AI Chatbot Performance Against Prosthodontic Residents on the National Prosthodontic Resident Examination (NPRE) from 2011 to 2023},
  journal = {The International Journal of Prosthodontics},
  year = {2025},
  doi = {10.11607/ijp.9486}
}
Start for free →