EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Multidimensional evaluation of large language models on the AAP in-service examination: Assessing accuracy, calibration, and citation reliability

Prita Abhay Dhaimade iD, Robin Henderson

PLOS Digital Health · 29.05.2026 · 5 (5) · e0001072

DOI: 10.1371/journal.pdig.0001072 ↗

Kurz gefasst

Fachartikel in der Zeitschrift PLOS Digital Health, online erschienen am 29.05.2026. Verfasst von Prita Abhay Dhaimade und 1 weiteren Autor:innen. Beteiligte Einrichtung: Division of Periodontology, Department of Surgical Sciences, School of Dental Medicine, East Carolina University, Greenville, North Carolina, United …. Die Arbeit wurde bisher 2-mal zitiert (Crossref) und nennt 26 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Large language models (LLMs) have demonstrated rapid advancements in natural language understanding and generation, prompting their integration into biomedical research, clinical practice, and professional education. However, systematic evaluation of LLMs in specialty-specific domains such as dentistry and periodontology remains limited, particularly regarding multidimensional performance metrics. This study conducted a comprehensive assessment of commercially available LLMs — GPT-4.0, GPT-5.0, and Claude Sonnet 4.0 — on the American Academy of Periodontology In-Service Examination, focusing on response accuracy, self-assessed confidence calibration, citation validity, and hallucination prevalence. Models were evaluated on the 2024 AAP In-Service Examination (331 questions) using two formats: Full Test (all questions at once) and Individual Question (one at a time). Prompts were standardized; models selected answers, and GPT-5.0 and Claude Sonnet 4.0 also provided confidence ratings and citations. Citation validity was assessed using a human-in-the-loop protocol with expert review. Statistical analyses included chi-square, McNemar’s, and logistic regression to assess accuracy, question fatigue, confidence calibration, and citation reliability. LLMs achieved high overall accuracy (78–87%), with the Individual Question format consistently yielding higher scores than Full Test, though differences were not statistically significant. Accuracy was highest in fact-dense domains (biochemistry, physiology, microbiology) and lowest in integrative domains (diagnosis, therapy). Significant question fatigue was observed in GPT-5.0 Full Test mode (OR = 0.997, p = 0.035) but not in Individual Question mode. Confidence scores predicted accuracy, with the strongest calibration in Individual Question mode. Citation analysis revealed frequent hallucinations, mostly critically erroneous, and citation validity was independent of answer accuracy. LLMs show promise as adjunctive tools for periodontal education, but their outputs, especially for complex reasoning and citations require rigorous human review to ensure accuracy and safety.

Autor:innen und Institutionen

  • Division of Periodontology, Department of Surgical Sciences, School of Dental Medicine, East Carolina University, Greenville, North Carolina, United States of America.
  • Division of Periodontics, College of Dentistry, University of Oklahoma Health Sciences Center, Oklahoma City, Oklahoma, United States of America.

ORCID-Profile: Prita Abhay Dhaimade

Zitieren

APA

Dhaimade, P. A., & Henderson, R. (2026). Multidimensional evaluation of large language models on the AAP in-service examination: Assessing accuracy, calibration, and citation reliability. PLOS Digital Health, 5(5), e0001072. https://doi.org/10.1371/journal.pdig.0001072

Vancouver

Dhaimade PA, Henderson R. Multidimensional evaluation of large language models on the AAP in-service examination: Assessing accuracy, calibration, and citation reliability. PLOS Digital Health. 2026;5(5):e0001072. doi:10.1371/journal.pdig.0001072

BibTeX
@article{Dhaimade2026,
  author = {Dhaimade, Prita Abhay and Henderson, Robin},
  title = {Multidimensional evaluation of large language models on the AAP in-service examination: Assessing accuracy, calibration, and citation reliability},
  journal = {PLOS Digital Health},
  year = {2026},
  volume = {5},
  number = {5},
  pages = {e0001072},
  doi = {10.1371/journal.pdig.0001072}
}
Kostenlos starten →