Accuracy of Large Language Models for Infective Endocarditis Prophylaxis in Dental Procedures
International Dental Journal · 2025
DOI: 10.1016/j.identj.2024.09.033 ↗Kurz gefasst
Fachartikel in der Zeitschrift International Dental Journal, 2025. Verfasst von Paak Rewthamrongsris und 3 weiteren Autor:innen. Beteiligte Einrichtung: Department of Anatomy, Faculty of Dentistry, Chulalongkorn University, Bangkok, Thailand. Hauptthemen laut MEDLINE: Antibiotic Prophylaxis, Endocarditis, Dental Care, Endocarditis, Bacterial. Die Arbeit wurde bisher 39-mal zitiert (Crossref) und nennt 31 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-NC-ND 4.0). Gefördert von: Ministry of Higher Education, Science, Research and Innovation, Thailand, Faculty of Dentistry, Chulalongkorn University, Thailand Science Research and Innovation.
Abstract (Originalsprache)
PURPOSE: Infective endocarditis (IE) is a serious, life-threatening condition requiring antibiotic prophylaxis for high-risk individuals undergoing invasive dental procedures. As LLMs are rapidly adopted by dental professionals for their efficiency and accessibility, assessing their accuracy in answering critical questions about antibiotic prophylaxis for IE prevention is crucial. METHODS: Twenty-eight true/false questions based on the 2021 American Heart Association (AHA) guidelines for IE were posed to 7 popular LLMs. Each model underwent five independent runs per question using two prompt strategies: a pre-prompt as an experienced dentist and without a pre-prompt. Inter-model comparisons utilised the Kruskal-Wallis test, followed by post-hoc pairwise comparisons using Prism 10 software. RESULTS: Significant differences in accuracy were observed among the LLMs. All LLMs had a narrower confidence interval with a pre-prompt, and most, except Claude 3 Opus, showed improved performance. GPT-4o had the highest accuracy (80% with a pre-prompt, 78.57% without), followed by Gemini 1.5 Pro (78.57% and 77.86%) and Claude 3 Opus (75.71% and 77.14%). Gemini 1.5 Flash had the lowest accuracy (68.57% and 63.57%). Without a pre-prompt, Gemini 1.5 Flash's accuracy was significantly lower than Claude 3 Opus, Gemini 1.5 Pro, and GPT-4o. With a pre-prompt, Gemini 1.5 Flash and Claude 3.5 were significantly less accurate than Gemini 1.5 Pro and GPT-4o. None of the LLMs met the commonly used benchmark scores. All models provided both correct and incorrect answers randomly, except Claude 3.5 Sonnet with a pre-prompt, which consistently gave incorrect answers to eight questions across five runs. CONCLUSION: LLMs like GPT-4o show promise for retrieving AHA-IE guideline information, achieving up to 80% accuracy. However, complex medical questions may still pose a challenge. Pre-prompts offer a potential solution, and domain-specific training is essential for optimizing LLM performance in healthcare, especially with the emergence of models with increased token limits.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Department of Anatomy, Faculty of Dentistry, Chulalongkorn University, Bangkok, Thailand.
- Stanford University, Stanford, California, USA.
- Department of Oral and Maxillofacial Surgery, Faculty of Dentistry, Chulalongkorn University, Bangkok, Thailand.
- Center of Excellence in Genomics and Precision Dentistry, Clinical Research Center, Geriatric Dentistry and Special Patients Care International Program, Department of Physiology, Faculty of Dentistry, Chulalongkorn Univ…
Förderung
- Ministry of Higher Education, Science, Research and Innovation, Thailand
- Faculty of Dentistry, Chulalongkorn University
- Thailand Science Research and Innovation
- Chulalongkorn University
- National Research Council of Thailand
- Health Systems Research Institute
Zitieren
APA
Rewthamrongsris, P., Burapacheep, J., Trachoo, V., & Porntaveetus, T. (2025). Accuracy of Large Language Models for Infective Endocarditis Prophylaxis in Dental Procedures. International Dental Journal. https://doi.org/10.1016/j.identj.2024.09.033
Vancouver
Rewthamrongsris P, Burapacheep J, Trachoo V, Porntaveetus T. Accuracy of Large Language Models for Infective Endocarditis Prophylaxis in Dental Procedures. International Dental Journal. 2025. doi:10.1016/j.identj.2024.09.033
BibTeX
@article{Rewthamrongsris2025,
author = {Rewthamrongsris, Paak and Burapacheep, Jirayu and Trachoo, Vorapat and Porntaveetus, Thantrira},
title = {Accuracy of Large Language Models for Infective Endocarditis Prophylaxis in Dental Procedures},
journal = {International Dental Journal},
year = {2025},
doi = {10.1016/j.identj.2024.09.033}
}
EUVIMED