EUVIMEDEUVIMED entdecken
Mixed-Methods-StudieOpen Access

Assessing ChatGPT’s Mastery of Bloom’s Taxonomy Using Psychosomatic Medicine Exam Questions: Mixed-Methods Study

Anne Herrmann-Werner iD, Teresa Festl-Wietek iD, Friederike Holderried iD, Lea Herschbach iD, Jan Griewatz iD, Ken Masters iD, Stephan Zipfel iD, Moritz Mahling iD

Journal of Medical Internet Research · 23.01.2024 · 26 · e52113

DOI: 10.2196/52113 ↗

Kurz gefasst

Mixed-Methods-Studie in der Zeitschrift Journal of Medical Internet Research, online erschienen am 23.01.2024. Verfasst von Anne Herrmann-Werner und 7 weiteren Autor:innen. Beteiligte Einrichtung: Tübingen Institute for Medical Education, Faculty of Medicine, University of Tübingen, Tübingen, Germany. Hauptthemen laut MEDLINE: Education, Medical, Medicine, Psychosomatic Medicine. Die Arbeit wurde bisher 100-mal zitiert (Crossref) und nennt 36 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

Large language models such as GPT-4 (Generative Pre-trained Transformer 4) are being increasingly used in medicine and medical education. However, these models are prone to “hallucinations” (ie, outputs that seem convincing while being factually incorrect). It is currently unknown how these errors by large language models relate to the different cognitive levels defined in Bloom’s taxonomy.

Objective

This study aims to explore how GPT-4 performs in terms of Bloom’s taxonomy using psychosomatic medicine exam questions.

Methods

We used a large data set of psychosomatic medicine multiple-choice questions (N=307) with real-world results derived from medical school exams. GPT-4 answered the multiple-choice questions using 2 distinct prompt versions: detailed and short. The answers were analyzed using a quantitative approach and a qualitative approach. Focusing on incorrectly answered questions, we categorized reasoning errors according to the hierarchical framework of Bloom’s taxonomy.

Results

GPT-4’s performance in answering exam questions yielded a high success rate: 93% (284/307) for the detailed prompt and 91% (278/307) for the short prompt. Questions answered correctly by GPT-4 had a statistically significant higher difficulty than questions answered incorrectly (P=.002 for the detailed prompt and P

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Education, MedicalMedicinePsychosomatic MedicineHumansResearch Design

Stichwörter der Autor:innen

Bloom’s taxonomyChatGPTGPT-4Generative Pre-trained Transformer 4LLMMCQNLPanswerartificial intelligenceassessmentclassificationerrorexamexaminationgenerative

Autor:innen und Institutionen

  • Tübingen Institute for Medical Education, Faculty of Medicine, University of Tübingen, Tübingen, Germany.
  • Department of Psychosomatic Medicine and Psychotherapy, University Hospital Tübingen, Tübingen, Germany.
  • University Department of Anesthesiology and Intensive Care Medicine, University Hospital Tübingen, Tübingen, Germany.
  • Medical Education and Informatics Department, College of Medicine and Health Sciences, Sultan Qaboos University, Muscat, Oman.
  • Department of Diabetology, Endocrinology, Nephrology, Section of Nephrology and Hypertension, University Hospital Tübingen, Tübingen, Germany.

ORCID-Profile: Anne Herrmann-Werner, Teresa Festl-Wietek, Friederike Holderried, Lea Herschbach, Jan Griewatz, Ken Masters, Stephan Zipfel, Moritz Mahling

Zitieren

APA

Herrmann-Werner, A., Festl-Wietek, T., Holderried, F., Herschbach, L., Griewatz, J., Masters, K., Zipfel, S., & Mahling, M. (2024). Assessing ChatGPT’s Mastery of Bloom’s Taxonomy Using Psychosomatic Medicine Exam Questions: Mixed-Methods Study. Journal of Medical Internet Research, 26, e52113. https://doi.org/10.2196/52113

Vancouver

Herrmann-Werner A, Festl-Wietek T, Holderried F, Herschbach L, Griewatz J, Masters K, et al. Assessing ChatGPT’s Mastery of Bloom’s Taxonomy Using Psychosomatic Medicine Exam Questions: Mixed-Methods Study. Journal of Medical Internet Research. 2024;26:e52113. doi:10.2196/52113

BibTeX
@article{HerrmannWerner2024,
  author = {Herrmann-Werner, Anne and Festl-Wietek, Teresa and Holderried, Friederike and Herschbach, Lea and Griewatz, Jan and Masters, Ken and Zipfel, Stephan and Mahling, Moritz},
  title = {Assessing ChatGPT’s Mastery of Bloom’s Taxonomy Using Psychosomatic Medicine Exam Questions: Mixed-Methods Study},
  journal = {Journal of Medical Internet Research},
  year = {2024},
  volume = {26},
  pages = {e52113},
  doi = {10.2196/52113}
}
Kostenlos starten →