EUVIMEDEUVIMED entdecken
VergleichsstudieOpen Access · CC BY-NC-ND 4.0

Performance of three artificial intelligence (AI)‐based large language models in standardized testing; implications for AI‐assisted dental education

Hamoun Sabri iD, Muhammad H A Saleh, Parham Hazrati iD, Keith Merchant, Jonathan Misch, Purnima S Kumar iD, Hom-Lay Wang iD, Shayan Barootchi iD

Journal of Periodontal Research · 2025-02 · 60 (2) · 121-133

DOI: 10.1111/jre.13323 ↗

Kurz gefasst

Vergleichsstudie in der Zeitschrift Journal of Periodontal Research, online erschienen am 18.07.2024. Verfasst von Hamoun Sabri und 7 weiteren Autor:innen. Beteiligte Einrichtung: Department of Periodontics and Oral Medicine, School of Dentistry, University of Michigan, Ann Arbor, Michigan, USA. Hauptthemen laut MEDLINE: Artificial Intelligence, Education, Dental, Educational Measurement, Periodontics. Die Arbeit wurde bisher 75-mal zitiert (Crossref) und nennt 31 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-NC-ND 4.0).

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Introduction

The emerging rise in novel computer technologies and automated data analytics has the potential to change the course of dental education. In line with our long‐term goal of harnessing the power of AI to augment didactic teaching, the objective of this study was to quantify and compare the accuracy of responses provided by ChatGPT (GPT‐4 and GPT‐3.5) and Google Gemini, the three primary large language models (LLMs), to human graduate students (control group) to the annual in‐service examination questions posed by the American Academy of Periodontology (AAP).

Methods

Under a comparative cross‐sectional study design, a corpus of 1312 questions from the annual in‐service examination of AAP administered between 2020 and 2023 were presented to the LLMs. Their responses were analyzed using chi‐square tests, and the performance was juxtaposed to the scores of periodontal residents from corresponding years, as the human control group. Additionally, two sub‐analyses were performed: one on the performance of the LLMs on each section of the exam; and in answering the most difficult questions.

Results

ChatGPT‐4 (total average: 79.57%) outperformed all human control groups as well as GPT‐3.5 and Google Gemini in all exam years ( p < .001). This chatbot showed an accuracy range between 78.80% and 80.98% across the various exam years. Gemini consistently recorded superior performance with scores of 70.65% ( p = .01), 73.29% ( p = .02), 75.73% ( p < .01), and 72.18% ( p = .0008) for the exams from 2020 to 2023 compared to ChatGPT‐3.5, which achieved 62.5%, 68.24%, 69.83%, and 59.27% respectively. Google Gemini (72.86%) surpassed the average scores achieved by first‐ (63.48% ± 31.67) and second‐year residents (66.25% ± 31.61) when all exam years combined. However, it could not surpass that of third‐year residents (69.06% ± 30.45).

Conclusions

Within the confines of this analysis, ChatGPT‐4 exhibited a robust capability in answering AAP in‐service exam questions in terms of accuracy and reliability while Gemini and ChatGPT‐3.5 showed a weaker performance. These findings underscore the potential of deploying LLMs as an educational tool in periodontics and oral implantology domains. However, the current limitations of these models such as inability to effectively process image‐based inquiries, the propensity for generating inconsistent responses to the same prompts, and achieving high (80% by GPT‐4) but not absolute accuracy rates should be considered. An objective comparison of their capability versus their capacity is required to further develop this field of study.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Artificial IntelligenceEducation, DentalEducational MeasurementPeriodonticsHumansCross-Sectional StudiesLarge Language Models

Stichwörter der Autor:innen

American Academy of PeriodontologyChatGPTChatGPT‐3.5ChatGPT‐4GeminiGoogle BardGoogle Geminiartificial intelligencedental educationperiodontal education

Autor:innen und Institutionen

  • Department of Periodontics and Oral Medicine, School of Dentistry, University of Michigan, Ann Arbor, Michigan, USA.
  • Center for Clinical Research and Evidence Synthesis in Oral Tissue Regeneration (CRITERION), Ann Arbor, Michigan, USA.
  • Naval Post-Graduate Dental School, Bethesda, Maryland, USA.
  • Private Practice, Ann Arbor, Michigan, USA.
  • Division of Periodontology, Department of Oral Medicine, Infection, and Immunity, Harvard School of Dental Medicine, Boston, Massachusetts, USA.

ORCID-Profile: Hamoun Sabri, Parham Hazrati, Purnima S Kumar, Hom-Lay Wang, Shayan Barootchi

Zitieren

APA

Sabri, H., Saleh, M. H. A., Hazrati, P., Merchant, K., Misch, J., Kumar, P. S., Wang, H. L., & Barootchi, S. (2025). Performance of three artificial intelligence (AI)‐based large language models in standardized testing; implications for AI‐assisted dental education. Journal of Periodontal Research, 60(2), 121-133. https://doi.org/10.1111/jre.13323

Vancouver

Sabri H, Saleh MHA, Hazrati P, Merchant K, Misch J, Kumar PS, et al. Performance of three artificial intelligence (AI)‐based large language models in standardized testing; implications for AI‐assisted dental education. Journal of Periodontal Research. 2025;60(2):121-133. doi:10.1111/jre.13323

BibTeX
@article{Sabri2025,
  author = {Sabri, Hamoun and Saleh, Muhammad H A and Hazrati, Parham and Merchant, Keith and Misch, Jonathan and Kumar, Purnima S and Wang, Hom-Lay and Barootchi, Shayan},
  title = {Performance of three artificial intelligence (AI)‐based large language models in standardized testing; implications for AI‐assisted dental education},
  journal = {Journal of Periodontal Research},
  year = {2025},
  volume = {60},
  number = {2},
  pages = {121--133},
  doi = {10.1111/jre.13323}
}
Kostenlos starten →