ChatGPT Generated Otorhinolaryngology Multiple‐Choice Questions: Quality, Psychometric Properties, and Suitability for Assessments
OTO Open · 2024-07 · 8 (3)
DOI: 10.1002/oto2.70018 ↗Kurz gefasst
Fachartikel in der Zeitschrift OTO Open, online erschienen am 26.09.2024. Verfasst von Cecilia Lotto und 6 weiteren Autor:innen. Beteiligte Einrichtung: Department of Otorhinolaryngology, Head and Neck Surgery Inselspital, Bern University Hospital, University of Bern Bern Switzerland. Die Arbeit wurde bisher 20-mal zitiert (Crossref) und nennt 16 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-NC-ND 4.0).
Abstract (Originalsprache)
Objective
To explore Chat Generative Pretrained Transformer's (ChatGPT's) capability to create multiple‐choice questions about otorhinolaryngology (ORL). Study Design Experimental question generation and exam simulation.
Setting
Tertiary academic center.
Methods
ChatGPT 3.5 was prompted: “Can you please create a challenging 20‐question multiple‐choice questionnaire about clinical cases in otolaryngology, offering five answer options?.” The generated questionnaire was sent to medical students, residents, and consultants. Questions were investigated regarding quality criteria. Answers were anonymized and the resulting data was analyzed in terms of difficulty and internal consistency.
Results
ChatGPT 3.5 generated 20 exam questions of which 1 question was considered off‐topic, 3 questions had a false answer, and 3 questions had multiple correct answers. Subspecialty theme repartition was as follows: 5 questions were on otology, 5 about rhinology, and 10 questions addressed head and neck. The qualities of focus and relevance were good while the vignette and distractor qualities were low. The level of difficulty was suitable for undergraduate medical students (n = 24), but too easy for residents (n = 30) or consultants (n = 10) in ORL. Cronbach's α was highest (.69) with 15 selected questions using students' results.
Conclusion
ChatGPT 3.5 is able to generate grammatically correct simple ORL multiple choice questions for a medical student level. However, the overall quality of the questions was average, needing thorough review and revision by a medical expert to ensure suitability in future exams.
Schlagwörter
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Department of Otorhinolaryngology, Head and Neck Surgery Inselspital, Bern University Hospital, University of Bern Bern Switzerland.
- Department of Otolaryngology, Head and Neck Surgery IRCCS Azienda Ospedaliero-Universitaria di Bologna Bologna Italy.
- Department of Medical and Surgical Sciences Alma Mater Studiorum-University of Bologna Bologna Italy.
- Institute for Medical Education University of Bern Bern Switzerland.
- Department of Otorhinolaryngology, Head and Neck Surgery CHUV, University of Lausanne Lausanne Switzerland.
- The Sense Innovation and Research Center Lausanne Switzerland.
ORCID-Profile: Cecilia Lotto, Sean C Sheppard, Wilma Anschuetz, Daniel Stricker, Giulia Molinari, Sören Huwendiek, Lukas Anschuetz
Zitieren
APA
Lotto, C., Sheppard, S. C., Anschuetz, W., Stricker, D., Molinari, G., Huwendiek, S., & Anschuetz, L. (2024). ChatGPT Generated Otorhinolaryngology Multiple‐Choice Questions: Quality, Psychometric Properties, and Suitability for Assessments. OTO Open, 8(3). https://doi.org/10.1002/oto2.70018
Vancouver
Lotto C, Sheppard SC, Anschuetz W, Stricker D, Molinari G, Huwendiek S, et al. ChatGPT Generated Otorhinolaryngology Multiple‐Choice Questions: Quality, Psychometric Properties, and Suitability for Assessments. OTO Open. 2024;8(3). doi:10.1002/oto2.70018
BibTeX
@article{Lotto2024,
author = {Lotto, Cecilia and Sheppard, Sean C and Anschuetz, Wilma and Stricker, Daniel and Molinari, Giulia and Huwendiek, Sören and Anschuetz, Lukas},
title = {ChatGPT Generated Otorhinolaryngology Multiple‐Choice Questions: Quality, Psychometric Properties, and Suitability for Assessments},
journal = {OTO Open},
year = {2024},
volume = {8},
number = {3},
doi = {10.1002/oto2.70018}
}
EUVIMED