EUVIMEDEUVIMED entdecken
Open Access · CC BY-NC-ND 4.0

ChatGPT Generated Otorhinolaryngology Multiple‐Choice Questions: Quality, Psychometric Properties, and Suitability for Assessments

Cecilia Lotto iD, Sean C Sheppard iD, Wilma Anschuetz iD, Daniel Stricker iD, Giulia Molinari iD, Sören Huwendiek iD, Lukas Anschuetz iD

OTO Open · 2024-07 · 8 (3)

DOI: 10.1002/oto2.70018 ↗

Kurz gefasst

Fachartikel in der Zeitschrift OTO Open, online erschienen am 26.09.2024. Verfasst von Cecilia Lotto und 6 weiteren Autor:innen. Beteiligte Einrichtung: Department of Otorhinolaryngology, Head and Neck Surgery Inselspital, Bern University Hospital, University of Bern Bern Switzerland. Die Arbeit wurde bisher 20-mal zitiert (Crossref) und nennt 16 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-NC-ND 4.0).

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Objective

To explore Chat Generative Pretrained Transformer's (ChatGPT's) capability to create multiple‐choice questions about otorhinolaryngology (ORL). Study Design Experimental question generation and exam simulation.

Setting

Tertiary academic center.

Methods

ChatGPT 3.5 was prompted: “Can you please create a challenging 20‐question multiple‐choice questionnaire about clinical cases in otolaryngology, offering five answer options?.” The generated questionnaire was sent to medical students, residents, and consultants. Questions were investigated regarding quality criteria. Answers were anonymized and the resulting data was analyzed in terms of difficulty and internal consistency.

Results

ChatGPT 3.5 generated 20 exam questions of which 1 question was considered off‐topic, 3 questions had a false answer, and 3 questions had multiple correct answers. Subspecialty theme repartition was as follows: 5 questions were on otology, 5 about rhinology, and 10 questions addressed head and neck. The qualities of focus and relevance were good while the vignette and distractor qualities were low. The level of difficulty was suitable for undergraduate medical students (n = 24), but too easy for residents (n = 30) or consultants (n = 10) in ORL. Cronbach's α was highest (.69) with 15 selected questions using students' results.

Conclusion

ChatGPT 3.5 is able to generate grammatically correct simple ORL multiple choice questions for a medical student level. However, the overall quality of the questions was average, needing thorough review and revision by a medical expert to ensure suitability in future exams.

Schlagwörter

Stichwörter der Autor:innen

ChatGPTartificial intelligenceexamlarge language modelmultiple choice questionotolaryngology

Autor:innen und Institutionen

  • Department of Otorhinolaryngology, Head and Neck Surgery Inselspital, Bern University Hospital, University of Bern Bern Switzerland.
  • Department of Otolaryngology, Head and Neck Surgery IRCCS Azienda Ospedaliero-Universitaria di Bologna Bologna Italy.
  • Department of Medical and Surgical Sciences Alma Mater Studiorum-University of Bologna Bologna Italy.
  • Institute for Medical Education University of Bern Bern Switzerland.
  • Department of Otorhinolaryngology, Head and Neck Surgery CHUV, University of Lausanne Lausanne Switzerland.
  • The Sense Innovation and Research Center Lausanne Switzerland.

ORCID-Profile: Cecilia Lotto, Sean C Sheppard, Wilma Anschuetz, Daniel Stricker, Giulia Molinari, Sören Huwendiek, Lukas Anschuetz

Zitieren

APA

Lotto, C., Sheppard, S. C., Anschuetz, W., Stricker, D., Molinari, G., Huwendiek, S., & Anschuetz, L. (2024). ChatGPT Generated Otorhinolaryngology Multiple‐Choice Questions: Quality, Psychometric Properties, and Suitability for Assessments. OTO Open, 8(3). https://doi.org/10.1002/oto2.70018

Vancouver

Lotto C, Sheppard SC, Anschuetz W, Stricker D, Molinari G, Huwendiek S, et al. ChatGPT Generated Otorhinolaryngology Multiple‐Choice Questions: Quality, Psychometric Properties, and Suitability for Assessments. OTO Open. 2024;8(3). doi:10.1002/oto2.70018

BibTeX
@article{Lotto2024,
  author = {Lotto, Cecilia and Sheppard, Sean C and Anschuetz, Wilma and Stricker, Daniel and Molinari, Giulia and Huwendiek, Sören and Anschuetz, Lukas},
  title = {ChatGPT Generated Otorhinolaryngology Multiple‐Choice Questions: Quality, Psychometric Properties, and Suitability for Assessments},
  journal = {OTO Open},
  year = {2024},
  volume = {8},
  number = {3},
  doi = {10.1002/oto2.70018}
}
Kostenlos starten →