Artificial intelligence performance in answering multiple-choice oral pathology questions: a comparative analysis
BMC Oral Health · 15.04.2025 · 25 (1)
DOI: 10.1186/s12903-025-05926-2 ↗Kurz gefasst
Vergleichsstudie in der Zeitschrift BMC Oral Health, online erschienen am 15.04.2025. Verfasst von Birkan Eyup Yilmaz und 2 weiteren Autor:innen. Beteiligte Einrichtung: Faculty of Dentistry, Department of Oral and Maxillofacial Surgery, Giresun University, Giresun, Türkiye. ylmzbirkan@gmail.com. Hauptthemen laut MEDLINE: Artificial Intelligence, Pathology, Oral, Educational Measurement, Education, Dental. Die Arbeit wurde bisher 77-mal zitiert (Crossref) und nennt 22 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-NC-ND 4.0).
Abstract (Originalsprache)
BACKGROUND: Artificial intelligence (AI) has rapidly advanced in healthcare and dental education, significantly impacting diagnostic processes, treatment planning, and academic training. The aim of this study is to evaluate the performance differences between different large language models (LLMs) by analyzing their accuracy rates in answers to multiple choice oral pathology questions. METHODS: This study evaluates the performance of eight LLMs (Gemini 1.5, Gemini 2, ChatGPT 4o, ChatGPT 4, ChatGPT o1, Copilot, Claude 3.5, Deepseek) in answering multiple-choice oral pathology questions from the Turkish Dental Specialization Examination (DUS). A total of 100 questions from 2012 to 2021 were analyzed. Questions were classified as "case-based" or "knowledge-based". The responses were classified as "correct" or "incorrect" based on official answer keys. To prevent learning biases, no follow-up questions or feedback were provided after the LLMs' responses. RESULTS: Significant performance differences were observed among the models (p < 0.001). ChatGPT o1 achieved the highest accuracy (96 correct, 4 incorrect), followed by Claude (84 correct), Gemini 2 and Deepseek (82 correct each). Copilot had the lowest performance (61 correct). Case-based questions showed notable performance variations (p = 0.034), where ChatGPT o1 and Claude excelled. For knowledge-based questions, ChatGPT o1 and Deepseek demonstrated the highest accuracy (p < 0.001). Post-hoc analysis revealed that ChatGPT o1 performed significantly better than most other models across both case-based and knowledge-based questions (p < 0.0031). CONCLUSION: LLMs demonstrated variable proficiency in oral pathology questions, with ChatGPT o1 showing higher accuracy. LLMs shows promise as a supplementary educational tool, though further validation is required.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Faculty of Dentistry, Department of Oral and Maxillofacial Surgery, Giresun University, Giresun, Türkiye. ylmzbirkan@gmail.com.
- Giresun Oral and Dental Health Centre, Department of Dentomaxillofacial Radiology, Giresun, Türkiye.
- Faculty of Dentistry, Department of Dentomaxillofacial Radiology, Afyonkarahisar Health Sciences University, Afyonkarahisar, Türkiye.
Zitieren
APA
Yilmaz, B. E., Gokkurt Yilmaz, B. N., & Ozbey, F. (2025). Artificial intelligence performance in answering multiple-choice oral pathology questions: a comparative analysis. BMC Oral Health, 25(1). https://doi.org/10.1186/s12903-025-05926-2
Vancouver
Yilmaz BE, Gokkurt Yilmaz BN, Ozbey F. Artificial intelligence performance in answering multiple-choice oral pathology questions: a comparative analysis. BMC Oral Health. 2025;25(1). doi:10.1186/s12903-025-05926-2
BibTeX
@article{Yilmaz2025,
author = {Yilmaz, Birkan Eyup and Gokkurt Yilmaz, Busra Nur and Ozbey, Furkan},
title = {Artificial intelligence performance in answering multiple-choice oral pathology questions: a comparative analysis},
journal = {BMC Oral Health},
year = {2025},
volume = {25},
number = {1},
doi = {10.1186/s12903-025-05926-2}
}
EUVIMED