EUVIMEDOpdag EUVIMED
Mixed-Methods-StudieOpen access

Assessing ChatGPT’s Mastery of Bloom’s Taxonomy Using Psychosomatic Medicine Exam Questions: Mixed-Methods Study

Anne Herrmann-Werner iD, Teresa Festl-Wietek iD, Friederike Holderried iD, Lea Herschbach iD, Jan Griewatz iD, Ken Masters iD, Stephan Zipfel iD, Moritz Mahling iD

Journal of Medical Internet Research · 2024-01-23 · 26 · e52113

DOI: 10.2196/52113 ↗

At a glance

Mixed-Methods-Studie in Journal of Medical Internet Research, published online on 2024-01-23. Written by Anne Herrmann-Werner and 7 co-authors. Institution: Tübingen Institute for Medical Education, Faculty of Medicine, University of Tübingen, Tübingen, Germany. Major MEDLINE topics: Education, Medical, Medicine, Psychosomatic Medicine. Cited 100 times so far (Crossref); 36 references. The full text is freely available in PubMed Central.

AbstractSubjectsBibliografiske oplysningerCiteRelated studies

Abstract (original language)

Background

Large language models such as GPT-4 (Generative Pre-trained Transformer 4) are being increasingly used in medicine and medical education. However, these models are prone to “hallucinations” (ie, outputs that seem convincing while being factually incorrect). It is currently unknown how these errors by large language models relate to the different cognitive levels defined in Bloom’s taxonomy.

Objective

This study aims to explore how GPT-4 performs in terms of Bloom’s taxonomy using psychosomatic medicine exam questions.

Methods

We used a large data set of psychosomatic medicine multiple-choice questions (N=307) with real-world results derived from medical school exams. GPT-4 answered the multiple-choice questions using 2 distinct prompt versions: detailed and short. The answers were analyzed using a quantitative approach and a qualitative approach. Focusing on incorrectly answered questions, we categorized reasoning errors according to the hierarchical framework of Bloom’s taxonomy.

Results

GPT-4’s performance in answering exam questions yielded a high success rate: 93% (284/307) for the detailed prompt and 91% (278/307) for the short prompt. Questions answered correctly by GPT-4 had a statistically significant higher difficulty than questions answered incorrectly (P=.002 for the detailed prompt and P

Subjects

MeSH terms (MEDLINE)

Education, MedicalMedicinePsychosomatic MedicineHumansResearch Design

Author keywords

Bloom’s taxonomyChatGPTGPT-4Generative Pre-trained Transformer 4LLMMCQNLPanswerartificial intelligenceassessmentclassificationerrorexamexaminationgenerative

Authors and institutions

  • Tübingen Institute for Medical Education, Faculty of Medicine, University of Tübingen, Tübingen, Germany.
  • Department of Psychosomatic Medicine and Psychotherapy, University Hospital Tübingen, Tübingen, Germany.
  • University Department of Anesthesiology and Intensive Care Medicine, University Hospital Tübingen, Tübingen, Germany.
  • Medical Education and Informatics Department, College of Medicine and Health Sciences, Sultan Qaboos University, Muscat, Oman.
  • Department of Diabetology, Endocrinology, Nephrology, Section of Nephrology and Hypertension, University Hospital Tübingen, Tübingen, Germany.

ORCID profiles: Anne Herrmann-Werner, Teresa Festl-Wietek, Friederike Holderried, Lea Herschbach, Jan Griewatz, Ken Masters, Stephan Zipfel, Moritz Mahling

Cite

APA

Herrmann-Werner, A., Festl-Wietek, T., Holderried, F., Herschbach, L., Griewatz, J., Masters, K., Zipfel, S., & Mahling, M. (2024). Assessing ChatGPT’s Mastery of Bloom’s Taxonomy Using Psychosomatic Medicine Exam Questions: Mixed-Methods Study. Journal of Medical Internet Research, 26, e52113. https://doi.org/10.2196/52113

Vancouver

Herrmann-Werner A, Festl-Wietek T, Holderried F, Herschbach L, Griewatz J, Masters K, et al. Assessing ChatGPT’s Mastery of Bloom’s Taxonomy Using Psychosomatic Medicine Exam Questions: Mixed-Methods Study. Journal of Medical Internet Research. 2024;26:e52113. doi:10.2196/52113

BibTeX
@article{HerrmannWerner2024,
  author = {Herrmann-Werner, Anne and Festl-Wietek, Teresa and Holderried, Friederike and Herschbach, Lea and Griewatz, Jan and Masters, Ken and Zipfel, Stephan and Mahling, Moritz},
  title = {Assessing ChatGPT’s Mastery of Bloom’s Taxonomy Using Psychosomatic Medicine Exam Questions: Mixed-Methods Study},
  journal = {Journal of Medical Internet Research},
  year = {2024},
  volume = {26},
  pages = {e52113},
  doi = {10.2196/52113}
}
Start for free →