Assessing ChatGPT’s Mastery of Bloom’s Taxonomy Using Psychosomatic Medicine Exam Questions: Mixed-Methods Study
Journal of Medical Internet Research · 23.01.2024 · 26 · e52113
DOI: 10.2196/52113 ↗Kurz gefasst
Mixed-Methods-Studie in der Zeitschrift Journal of Medical Internet Research, online erschienen am 23.01.2024. Verfasst von Anne Herrmann-Werner und 7 weiteren Autor:innen. Beteiligte Einrichtung: Tübingen Institute for Medical Education, Faculty of Medicine, University of Tübingen, Tübingen, Germany. Hauptthemen laut MEDLINE: Education, Medical, Medicine, Psychosomatic Medicine. Die Arbeit wurde bisher 100-mal zitiert (Crossref) und nennt 36 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.
Abstract (Originalsprache)
Background
Large language models such as GPT-4 (Generative Pre-trained Transformer 4) are being increasingly used in medicine and medical education. However, these models are prone to “hallucinations” (ie, outputs that seem convincing while being factually incorrect). It is currently unknown how these errors by large language models relate to the different cognitive levels defined in Bloom’s taxonomy.
Objective
This study aims to explore how GPT-4 performs in terms of Bloom’s taxonomy using psychosomatic medicine exam questions.
Methods
We used a large data set of psychosomatic medicine multiple-choice questions (N=307) with real-world results derived from medical school exams. GPT-4 answered the multiple-choice questions using 2 distinct prompt versions: detailed and short. The answers were analyzed using a quantitative approach and a qualitative approach. Focusing on incorrectly answered questions, we categorized reasoning errors according to the hierarchical framework of Bloom’s taxonomy.
Results
GPT-4’s performance in answering exam questions yielded a high success rate: 93% (284/307) for the detailed prompt and 91% (278/307) for the short prompt. Questions answered correctly by GPT-4 had a statistically significant higher difficulty than questions answered incorrectly (P=.002 for the detailed prompt and P
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Tübingen Institute for Medical Education, Faculty of Medicine, University of Tübingen, Tübingen, Germany.
- Department of Psychosomatic Medicine and Psychotherapy, University Hospital Tübingen, Tübingen, Germany.
- University Department of Anesthesiology and Intensive Care Medicine, University Hospital Tübingen, Tübingen, Germany.
- Medical Education and Informatics Department, College of Medicine and Health Sciences, Sultan Qaboos University, Muscat, Oman.
- Department of Diabetology, Endocrinology, Nephrology, Section of Nephrology and Hypertension, University Hospital Tübingen, Tübingen, Germany.
ORCID-Profile: Anne Herrmann-Werner, Teresa Festl-Wietek, Friederike Holderried, Lea Herschbach, Jan Griewatz, Ken Masters, Stephan Zipfel, Moritz Mahling
Zitieren
APA
Herrmann-Werner, A., Festl-Wietek, T., Holderried, F., Herschbach, L., Griewatz, J., Masters, K., Zipfel, S., & Mahling, M. (2024). Assessing ChatGPT’s Mastery of Bloom’s Taxonomy Using Psychosomatic Medicine Exam Questions: Mixed-Methods Study. Journal of Medical Internet Research, 26, e52113. https://doi.org/10.2196/52113
Vancouver
Herrmann-Werner A, Festl-Wietek T, Holderried F, Herschbach L, Griewatz J, Masters K, et al. Assessing ChatGPT’s Mastery of Bloom’s Taxonomy Using Psychosomatic Medicine Exam Questions: Mixed-Methods Study. Journal of Medical Internet Research. 2024;26:e52113. doi:10.2196/52113
BibTeX
@article{HerrmannWerner2024,
author = {Herrmann-Werner, Anne and Festl-Wietek, Teresa and Holderried, Friederike and Herschbach, Lea and Griewatz, Jan and Masters, Ken and Zipfel, Stephan and Mahling, Moritz},
title = {Assessing ChatGPT’s Mastery of Bloom’s Taxonomy Using Psychosomatic Medicine Exam Questions: Mixed-Methods Study},
journal = {Journal of Medical Internet Research},
year = {2024},
volume = {26},
pages = {e52113},
doi = {10.2196/52113}
}
EUVIMED