How well do large language model-based chatbots perform in oral and maxillofacial radiology?
Dentomaxillofacial Radiology · 01.09.2024 · 53 (6) · 390-395
DOI: 10.1093/dmfr/twae021 ↗Kurz gefasst
Fachartikel in der Zeitschrift Dentomaxillofacial Radiology, online erschienen am 07.06.2024. Verfasst von Hui Jeong und 4 weiteren Autor:innen. Beteiligte Einrichtung: Department of Oral and Maxillofacial Radiology, Yonsei University College of Dentistry, Seoul 03722, Republic of Korea. Hauptthemen laut MEDLINE: Radiology, Educational Measurement, Education, Dental. Die Arbeit wurde bisher 52-mal zitiert (Crossref) und nennt 22 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Yonsei University College of Dentistry Fund.
Abstract (Originalsprache)
Objectives
This study evaluated the performance of four large language model (LLM)-based chatbots by comparing their test results with those of dental students on an oral and maxillofacial radiology examination.
Methods
ChatGPT, ChatGPT Plus, Bard, and Bing Chat were tested on 52 questions from regular dental college examinations. These questions were categorized into three educational content areas: basic knowledge, imaging and equipment, and image interpretation. They were also classified as multiple-choice questions (MCQs) and short-answer questions (SAQs). The accuracy rates of the chatbots were compared with the performance of students, and further analysis was conducted based on the educational content and question type.
Results
The students’ overall accuracy rate was 81.2%, while that of the chatbots varied: 50.0% for ChatGPT, 65.4% for ChatGPT Plus, 50.0% for Bard, and 63.5% for Bing Chat. ChatGPT Plus achieved a higher accuracy rate for basic knowledge than the students (93.8% vs. 78.7%). However, all chatbots performed poorly in image interpretation, with accuracy rates below 35.0%. All chatbots scored less than 60.0% on MCQs, but performed better on SAQs.
Conclusions
The performance of chatbots in oral and maxillofacial radiology was unsatisfactory. Further training using specific, relevant data derived solely from reliable sources is required. Additionally, the validity of these chatbots’ responses must be meticulously verified.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Department of Oral and Maxillofacial Radiology, Yonsei University College of Dentistry, Seoul 03722, Republic of Korea.
- Institute for Innovation in Digital Healthcare, Yonsei University, Seoul 03722, Republic of Korea.
- Oral Science Research Center, College of Dentistry, Yonsei University, Seoul 03722, Republic of Korea.
- Department of Artificial Intelligence, Yonsei University College of Computing, Seoul 03722, Republic of Korea.
ORCID-Profile: Kug Jin Jeon
Förderung
- Yonsei University College of Dentistry Fund
Zitieren
APA
Jeong, H., Han, S. S., Yu, Y., Kim, S., & Jeon, K. J. (2024). How well do large language model-based chatbots perform in oral and maxillofacial radiology?. Dentomaxillofacial Radiology, 53(6), 390-395. https://doi.org/10.1093/dmfr/twae021
Vancouver
Jeong H, Han SS, Yu Y, Kim S, Jeon KJ. How well do large language model-based chatbots perform in oral and maxillofacial radiology?. Dentomaxillofacial Radiology. 2024;53(6):390-395. doi:10.1093/dmfr/twae021
BibTeX
@article{Jeong2024,
author = {Jeong, Hui and Han, Sang-Sun and Yu, Youngjae and Kim, Saejin and Jeon, Kug Jin},
title = {How well do large language model-based chatbots perform in oral and maxillofacial radiology?},
journal = {Dentomaxillofacial Radiology},
year = {2024},
volume = {53},
number = {6},
pages = {390--395},
doi = {10.1093/dmfr/twae021}
}
EUVIMED