Performance of large language models in oral and maxillofacial surgery examinations
International Journal of Oral and Maxillofacial Surgery · 2024-10 · 53 (10) · 881-886
DOI: 10.1016/j.ijom.2024.06.003 ↗Kurz gefasst
Fachartikel in der Zeitschrift International Journal of Oral and Maxillofacial Surgery, 2024-10. Verfasst von B Quah und 3 weiteren Autor:innen. Beteiligte Einrichtung: Faculty of Dentistry, National University of Singapore, Singapore; Discipline of Oral and Maxillofacial Surgery, National University Centre for Oral …. Hauptthemen laut MEDLINE: Surgery, Oral, Educational Measurement. Die Arbeit wurde bisher 44-mal zitiert (Crossref) und nennt 14 Literaturangaben.
Abstract (Originalsprache)
This study aimed to determine the accuracy of large language models (LLMs) in answering oral and maxillofacial surgery (OMS) multiple choice questions. A total of 259 questions from the university's question bank were answered by the LLMs (GPT-3.5, GPT-4, Llama 2, Gemini, and Copilot). The scores per category as well as the total score out of 259 were recorded and evaluated, with the passing score set at 50%. The mean overall score amongst all LLMs was 62.5%. GPT-4 performed the best (76.8%, 95% confidence interval (CI) 71.4-82.2%), followed by Copilot (72.6%, 95% CI 67.2-78.0%), GPT-3.5 (62.2%, 95% CI 56.4-68.0%), Gemini (58.7%, 95% CI 52.9-64.5%), and Llama 2 (42.5%, 95% CI 37.1-48.6%). There was a statistically significant difference between the scores of the five LLMs overall (χ2 = 79.9, df = 4, P < 0.001) and within all categories except 'basic sciences' (P = 0.129), 'dentoalveolar and implant surgery' (P = 0.052), and 'oral medicine/pathology/radiology' (P = 0.801). The LLMs performed best in 'basic sciences' (68.9%) and poorest in 'pharmacology' (45.9%). The LLMs can be used as adjuncts in teaching, but should not be used for clinical decision-making until the models are further developed and validated.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Faculty of Dentistry, National University of Singapore, Singapore; Discipline of Oral and Maxillofacial Surgery, National University Centre for Oral Health, Singapore.
- Faculty of Dentistry, National University of Singapore, Singapore.
- Faculty of Dentistry, National University of Singapore, Singapore; Discipline of Oral and Maxillofacial Surgery, National University Centre for Oral Health, Singapore. Electronic address: denii@nus.edu.sg.
Zitieren
APA
Quah, B., Yong, C. W., Lai, C. W. M., & Islam, I. (2024). Performance of large language models in oral and maxillofacial surgery examinations. International Journal of Oral and Maxillofacial Surgery, 53(10), 881-886. https://doi.org/10.1016/j.ijom.2024.06.003
Vancouver
Quah B, Yong CW, Lai CWM, Islam I. Performance of large language models in oral and maxillofacial surgery examinations. International Journal of Oral and Maxillofacial Surgery. 2024;53(10):881-886. doi:10.1016/j.ijom.2024.06.003
BibTeX
@article{Quah2024,
author = {Quah, B and Yong, C W and Lai, C W M and Islam, I},
title = {Performance of large language models in oral and maxillofacial surgery examinations},
journal = {International Journal of Oral and Maxillofacial Surgery},
year = {2024},
volume = {53},
number = {10},
pages = {881--886},
doi = {10.1016/j.ijom.2024.06.003}
}
EUVIMED