EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Evaluation of the accuracy and robustness of large language models in dental examinations: sensitivity analysis of question format and evaluation parameters

Andrea Atassi, Shachar Stewart, Carrie Brackenridge, Eliza Kamin

SAP Artificial Intelligence in Dentistry · 12.08.2026 · 2 · 135

DOI: 10.62486/aid2026135 ↗

Kurz gefasst

Fachartikel in der Zeitschrift SAP Artificial Intelligence in Dentistry, online erschienen am 12.08.2026. Verfasst von Andrea Atassi und 3 weiteren Autor:innen. Die Arbeit nennt 30 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Introduction

Large language models are increasingly evaluated on medical examination questions, but their performance on dental questions, and the robustness of that performance to question format and decoding settings, is not established.

Methods

A fixed set of 100 dental multiple-choice questions was drawn from the MedMCQA validation split and posed to a large language model in a zero-shot setting. Accuracy was scored against the reference answer, and sensitivity analyses varied question format, decoding temperature and sample size.

Results

The model answered 70 of 100 dental examination questions correctly, for an accuracy of 0.700 (95% CI 0.604 to 0.781). The estimate remained essentially unchanged at a higher decoding temperature (0.690) and on a 50-question subsample (0.700), while it differed between single-choice (0.733) and multiple-choice (0.625) question formats.

Conclusions

The model achieved a robust accuracy of 0.700 on dental examination questions, and the estimate was stable to temperature and sample size but sensitive to question format, with higher accuracy on single-choice than on multiple-choice questions.

Zitieren

APA

Atassi, A., Stewart, S., Brackenridge, C., & Kamin, E. (2026). Evaluation of the accuracy and robustness of large language models in dental examinations: sensitivity analysis of question format and evaluation parameters. SAP Artificial Intelligence in Dentistry, 2, 135. https://doi.org/10.62486/aid2026135

Vancouver

Atassi A, Stewart S, Brackenridge C, Kamin E. Evaluation of the accuracy and robustness of large language models in dental examinations: sensitivity analysis of question format and evaluation parameters. SAP Artificial Intelligence in Dentistry. 2026;2:135. doi:10.62486/aid2026135

BibTeX
@article{Atassi2026,
  author = {Atassi, Andrea and Stewart, Shachar and Brackenridge, Carrie and Kamin, Eliza},
  title = {Evaluation of the accuracy and robustness of large language models in dental examinations: sensitivity analysis of question format and evaluation parameters},
  journal = {SAP Artificial Intelligence in Dentistry},
  year = {2026},
  volume = {2},
  pages = {135},
  doi = {10.62486/aid2026135}
}
Kostenlos starten →