Assessing ChatGPT’s Mastery of Bloom’s Taxonomy Using Psychosomatic Medicine Exam Questions: Mixed-Methods Study
Journal of Medical Internet Research · 2024-01-23 · 26 · e52113
DOI: 10.2196/52113 ↗At a glance
Mixed-Methods-Studie in Journal of Medical Internet Research, published online on 2024-01-23. Written by Anne Herrmann-Werner and 7 co-authors. Institution: Tübingen Institute for Medical Education, Faculty of Medicine, University of Tübingen, Tübingen, Germany. Major MEDLINE topics: Education, Medical, Medicine, Psychosomatic Medicine. Cited 100 times so far (Crossref); 36 references. The full text is freely available in PubMed Central.
Abstract (original language)
Background
Large language models such as GPT-4 (Generative Pre-trained Transformer 4) are being increasingly used in medicine and medical education. However, these models are prone to “hallucinations” (ie, outputs that seem convincing while being factually incorrect). It is currently unknown how these errors by large language models relate to the different cognitive levels defined in Bloom’s taxonomy.
Objective
This study aims to explore how GPT-4 performs in terms of Bloom’s taxonomy using psychosomatic medicine exam questions.
Methods
We used a large data set of psychosomatic medicine multiple-choice questions (N=307) with real-world results derived from medical school exams. GPT-4 answered the multiple-choice questions using 2 distinct prompt versions: detailed and short. The answers were analyzed using a quantitative approach and a qualitative approach. Focusing on incorrectly answered questions, we categorized reasoning errors according to the hierarchical framework of Bloom’s taxonomy.
Results
GPT-4’s performance in answering exam questions yielded a high success rate: 93% (284/307) for the detailed prompt and 91% (278/307) for the short prompt. Questions answered correctly by GPT-4 had a statistically significant higher difficulty than questions answered incorrectly (P=.002 for the detailed prompt and P
Subjects
MeSH terms (MEDLINE)
Author keywords
Authors and institutions
- Tübingen Institute for Medical Education, Faculty of Medicine, University of Tübingen, Tübingen, Germany.
- Department of Psychosomatic Medicine and Psychotherapy, University Hospital Tübingen, Tübingen, Germany.
- University Department of Anesthesiology and Intensive Care Medicine, University Hospital Tübingen, Tübingen, Germany.
- Medical Education and Informatics Department, College of Medicine and Health Sciences, Sultan Qaboos University, Muscat, Oman.
- Department of Diabetology, Endocrinology, Nephrology, Section of Nephrology and Hypertension, University Hospital Tübingen, Tübingen, Germany.
ORCID profiles: Anne Herrmann-Werner, Teresa Festl-Wietek, Friederike Holderried, Lea Herschbach, Jan Griewatz, Ken Masters, Stephan Zipfel, Moritz Mahling
Cite
APA
Herrmann-Werner, A., Festl-Wietek, T., Holderried, F., Herschbach, L., Griewatz, J., Masters, K., Zipfel, S., & Mahling, M. (2024). Assessing ChatGPT’s Mastery of Bloom’s Taxonomy Using Psychosomatic Medicine Exam Questions: Mixed-Methods Study. Journal of Medical Internet Research, 26, e52113. https://doi.org/10.2196/52113
Vancouver
Herrmann-Werner A, Festl-Wietek T, Holderried F, Herschbach L, Griewatz J, Masters K, et al. Assessing ChatGPT’s Mastery of Bloom’s Taxonomy Using Psychosomatic Medicine Exam Questions: Mixed-Methods Study. Journal of Medical Internet Research. 2024;26:e52113. doi:10.2196/52113
BibTeX
@article{HerrmannWerner2024,
author = {Herrmann-Werner, Anne and Festl-Wietek, Teresa and Holderried, Friederike and Herschbach, Lea and Griewatz, Jan and Masters, Ken and Zipfel, Stephan and Mahling, Moritz},
title = {Assessing ChatGPT’s Mastery of Bloom’s Taxonomy Using Psychosomatic Medicine Exam Questions: Mixed-Methods Study},
journal = {Journal of Medical Internet Research},
year = {2024},
volume = {26},
pages = {e52113},
doi = {10.2196/52113}
}
EUVIMED