Evaluation of the accuracy and robustness of large language models in dental examinations: sensitivity analysis of question format and evaluation parameters
SAP Artificial Intelligence in Dentistry · 12.08.2026 · 2 · 135
DOI: 10.62486/aid2026135 ↗Kurz gefasst
Fachartikel in der Zeitschrift SAP Artificial Intelligence in Dentistry, online erschienen am 12.08.2026. Verfasst von Andrea Atassi und 3 weiteren Autor:innen. Die Arbeit nennt 30 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
Introduction
Large language models are increasingly evaluated on medical examination questions, but their performance on dental questions, and the robustness of that performance to question format and decoding settings, is not established.
Methods
A fixed set of 100 dental multiple-choice questions was drawn from the MedMCQA validation split and posed to a large language model in a zero-shot setting. Accuracy was scored against the reference answer, and sensitivity analyses varied question format, decoding temperature and sample size.
Results
The model answered 70 of 100 dental examination questions correctly, for an accuracy of 0.700 (95% CI 0.604 to 0.781). The estimate remained essentially unchanged at a higher decoding temperature (0.690) and on a 50-question subsample (0.700), while it differed between single-choice (0.733) and multiple-choice (0.625) question formats.
Conclusions
The model achieved a robust accuracy of 0.700 on dental examination questions, and the estimate was stable to temperature and sample size but sensitive to question format, with higher accuracy on single-choice than on multiple-choice questions.
Zitieren
APA
Atassi, A., Stewart, S., Brackenridge, C., & Kamin, E. (2026). Evaluation of the accuracy and robustness of large language models in dental examinations: sensitivity analysis of question format and evaluation parameters. SAP Artificial Intelligence in Dentistry, 2, 135. https://doi.org/10.62486/aid2026135
Vancouver
Atassi A, Stewart S, Brackenridge C, Kamin E. Evaluation of the accuracy and robustness of large language models in dental examinations: sensitivity analysis of question format and evaluation parameters. SAP Artificial Intelligence in Dentistry. 2026;2:135. doi:10.62486/aid2026135
BibTeX
@article{Atassi2026,
author = {Atassi, Andrea and Stewart, Shachar and Brackenridge, Carrie and Kamin, Eliza},
title = {Evaluation of the accuracy and robustness of large language models in dental examinations: sensitivity analysis of question format and evaluation parameters},
journal = {SAP Artificial Intelligence in Dentistry},
year = {2026},
volume = {2},
pages = {135},
doi = {10.62486/aid2026135}
}
EUVIMED