EUVIMEDEUVIMED entdecken
VergleichsstudieOpen Access · CC BY 4.0

GPT-4o’s competency in answering the simulated written European Board of Interventional Radiology exam compared to a medical student and experts in Germany and its ability to generate exam items on interventional radiology: a descriptive study

Sebastian Ebel, Constantin Ehrengut, Timm Denecke, Holger Gößmann, Anne Bettina Beeskow

Journal of Educational Evaluation for Health Professions · 20.08.2024 · 21 · 21

DOI: 10.3352/jeehp.2024.21.21 ↗

Kurz gefasst

Vergleichsstudie in der Zeitschrift Journal of Educational Evaluation for Health Professions, online erschienen am 20.08.2024. Verfasst von Sebastian Ebel und 4 weiteren Autor:innen. Beteiligte Einrichtung: Department of Diagnostic and Interventional Radiology, University of Leipzig, Leipzig, Germany. Hauptthemen laut MEDLINE: Educational Measurement, Radiology, Interventional, Clinical Competence, Students, Medical. Die Arbeit wurde bisher 18-mal zitiert (Crossref) und nennt 8 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Purpose

This study aimed to determine whether ChatGPT-4o, a generative artificial intelligence (AI) platform, was able to pass a simulated written European Board of Interventional Radiology (EBIR) exam and whether GPT-4o can be used to train medical students and interventional radiologists of different levels of expertise by generating exam items on interventional radiology.

Methods

GPT-4o was asked to answer 370 simulated exam items of the Cardiovascular and Interventional Radiology Society of Europe (CIRSE) for EBIR preparation (CIRSE Prep). Subsequently, GPT-4o was requested to generate exam items on interventional radiology topics at levels of difficulty suitable for medical students and the EBIR exam. Those generated items were answered by 4 participants, including a medical student, a resident, a consultant, and an EBIR holder. The correctly answered items were counted. One investigator checked the answers and items generated by GPT-4o for correctness and relevance. This work was done from April to July 2024.

Results

GPT-4o correctly answered 248 of the 370 CIRSE Prep items (67.0%). For 50 CIRSE Prep items, the medical student answered 46.0%, the resident 42.0%, the consultant 50.0%, and the EBIR holder 74.0% correctly. All participants answered 82.0% to 92.0% of the 50 GPT-4o generated items at the student level correctly. For the 50 GPT-4o items at the EBIR level, the medical student answered 32.0%, the resident 44.0%, the consultant 48.0%, and the EBIR holder 66.0% correctly. All participants could pass the GPT-4o-generated items for the student level; while the EBIR holder could pass the GPT-4o-generated items for the EBIR level. Two items (0.3%) out of 150 generated by the GPT-4o were assessed as implausible.

Conclusion

GPT-4o could pass the simulated written EBIR exam and create exam items of varying difficulty to train medical students and interventional radiologists.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Educational MeasurementRadiology, InterventionalClinical CompetenceStudents, MedicalHumansGermanyArtificial IntelligenceEurope

Stichwörter der Autor:innen

Artificial intelligenceConsultantsEuropeInterventional radiologyMedical students

Autor:innen und Institutionen

  • Department of Diagnostic and Interventional Radiology, University of Leipzig, Leipzig, Germany.

Zitieren

APA

Ebel, S., Ehrengut, C., Denecke, T., Gößmann, H., & Beeskow, A. B. (2024). GPT-4o’s competency in answering the simulated written European Board of Interventional Radiology exam compared to a medical student and experts in Germany and its ability to generate exam items on interventional radiology: a descriptive study. Journal of Educational Evaluation for Health Professions, 21, 21. https://doi.org/10.3352/jeehp.2024.21.21

Vancouver

Ebel S, Ehrengut C, Denecke T, Gößmann H, Beeskow AB. GPT-4o’s competency in answering the simulated written European Board of Interventional Radiology exam compared to a medical student and experts in Germany and its ability to generate exam items on interventional radiology: a descriptive study. Journal of Educational Evaluation for Health Professions. 2024;21:21. doi:10.3352/jeehp.2024.21.21

BibTeX
@article{Ebel2024,
  author = {Ebel, Sebastian and Ehrengut, Constantin and Denecke, Timm and Gößmann, Holger and Beeskow, Anne Bettina},
  title = {GPT-4o’s competency in answering the simulated written European Board of Interventional Radiology exam compared to a medical student and experts in Germany and its ability to generate exam items on interventional radiology: a descriptive study},
  journal = {Journal of Educational Evaluation for Health Professions},
  year = {2024},
  volume = {21},
  pages = {21},
  doi = {10.3352/jeehp.2024.21.21}
}
Kostenlos starten →