ChatGPT’s quiz skills in different otolaryngology subspecialties: an analysis of 2576 single-choice and multiple-choice board certification preparation questions
European Archives of Oto-Rhino-Laryngology · 2023
DOI: 10.1007/s00405-023-08051-4 ↗Kurz gefasst
Fachartikel in der Zeitschrift European Archives of Oto-Rhino-Laryngology, 2023. Verfasst von Cosima C Hoch und 7 weiteren Autor:innen. Beteiligte Einrichtung: Department of Otolaryngology, Head and Neck Surgery, School of Medicine, Technical University of Munich (TUM), Ismaningerstrasse 22, 81675, Munich, G…. Hauptthemen laut MEDLINE: Artificial Intelligence, Otolaryngology. Die Arbeit wurde bisher 160-mal zitiert (Crossref) und nennt 17 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Technische Universität München.
Abstract (Originalsprache)
Purpose
With the increasing adoption of artificial intelligence (AI) in various domains, including healthcare, there is growing acceptance and interest in consulting AI models to provide medical information and advice. This study aimed to evaluate the accuracy of ChatGPT’s responses to practice quiz questions designed for otolaryngology board certification and decipher potential performance disparities across different otolaryngology subspecialties.
Methods
A dataset covering 15 otolaryngology subspecialties was collected from an online learning platform funded by the German Society of Oto-Rhino-Laryngology, Head and Neck Surgery, designed for board certification examination preparation. These questions were entered into ChatGPT, with its responses being analyzed for accuracy and variance in performance.
Results
The dataset included 2576 questions (479 multiple-choice and 2097 single-choice), of which 57% ( n = 1475) were answered correctly by ChatGPT. An in-depth analysis of question style revealed that single-choice questions were associated with a significantly higher rate ( p < 0.001) of correct responses ( n = 1313; 63%) compared to multiple-choice questions ( n = 162; 34%). Stratified by question categories, ChatGPT yielded the highest rate of correct responses ( n = 151; 72%) in the field of allergology, whereas 7 out of 10 questions ( n = 65; 71%) on legal otolaryngology aspects were answered incorrectly.
Conclusion
The study reveals ChatGPT’s potential as a supplementary tool for otolaryngology board certification preparation. However, its propensity for errors in certain otolaryngology areas calls for further refinement. Future research should address these limitations to improve ChatGPT’s educational use. An approach, with expert collaboration, is recommended for the reliable and accurate integration of such AI models.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Department of Otolaryngology, Head and Neck Surgery, School of Medicine, Technical University of Munich (TUM), Ismaningerstrasse 22, 81675, Munich, Germany. cosima.chiara.hoch@tum.de.
- Department of Otolaryngology, Head and Neck Surgery, School of Medicine, Technical University of Munich (TUM), Ismaningerstrasse 22, 81675, Munich, Germany.
- Department of Otorhinolaryngology, Head and Neck Surgery, Medical Faculty, University of Cologne, 50937, Cologne, Germany.
- Division of Plastic Surgery, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, 02152, USA.
- Department of Plastic Surgery and Hand Surgery, Klinikum Rechts Der Isar, Technical University of Munich, Munich, Germany.
- Division of Plastic and Reconstructive Surgery, Massachusetts General Hospital, Harvard Medical School, Boston, MA, 02115, USA.
Förderung
- Technische Universität München
Zitieren
APA
Hoch, C. C., Wollenberg, B., Lüers, J. C., Knoedler, S., Knoedler, L., Frank, K., Cotofana, S., & Alfertshofer, M. (2023). ChatGPT’s quiz skills in different otolaryngology subspecialties: an analysis of 2576 single-choice and multiple-choice board certification preparation questions. European Archives of Oto-Rhino-Laryngology. https://doi.org/10.1007/s00405-023-08051-4
Vancouver
Hoch CC, Wollenberg B, Lüers JC, Knoedler S, Knoedler L, Frank K, et al. ChatGPT’s quiz skills in different otolaryngology subspecialties: an analysis of 2576 single-choice and multiple-choice board certification preparation questions. European Archives of Oto-Rhino-Laryngology. 2023. doi:10.1007/s00405-023-08051-4
BibTeX
@article{Hoch2023,
author = {Hoch, Cosima C and Wollenberg, Barbara and Lüers, Jan-Christoffer and Knoedler, Samuel and Knoedler, Leonard and Frank, Konstantin and Cotofana, Sebastian and Alfertshofer, Michael},
title = {ChatGPT’s quiz skills in different otolaryngology subspecialties: an analysis of 2576 single-choice and multiple-choice board certification preparation questions},
journal = {European Archives of Oto-Rhino-Laryngology},
year = {2023},
doi = {10.1007/s00405-023-08051-4}
}
EUVIMED