EUVIMEDEUVIMED entdecken
Open Access · CC BY-NC-ND 4.0

Evaluating GPT-4o in high-stakes medical assessments: performance and error analysis on a Chilean anesthesiology exam

Fernando R Altermatt, Andres Neyem, Nicolás I Sumonte, Ignacio Villagrán, Marcelo Mendoza, Hector J Lacassie, Alejandro E Delfino

BMC Medical Education · 27.10.2025 · 25 (1)

DOI: 10.1186/s12909-025-08084-9 ↗

Kurz gefasst

Fachartikel in der Zeitschrift BMC Medical Education, online erschienen am 27.10.2025. Verfasst von Fernando R Altermatt und 6 weiteren Autor:innen. Beteiligte Einrichtung: Division of Anesthesiology, School of Medicine, Pontificia Universidad Católica de Chile, Marcoleta 377, Santiago, Chile. falterma@uc.cl. Hauptthemen laut MEDLINE: Anesthesiology, Educational Measurement, Clinical Competence. Die Arbeit wurde bisher 13-mal zitiert (Crossref) und nennt 19 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-NC-ND 4.0). Gefördert von: Agencia Nacional de Investigación y Desarrollo, National Center for Artificial Intelligence.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

BACKGROUND: Large language models (LLMs) such as GPT-4o have the potential to transform clinical decision-making, patient education, and medical research. Despite impressive performance in generating patient-friendly educational materials and assisting in clinical documentation, concerns remain regarding the reliability, subtle errors, and biases that can undermine their use in high-stakes medical settings. METHODS: A multi-phase experimental design was employed to assess the performance of GPT-4o on the Chilean anesthesiology exam (CONACEM), which comprised 183 questions covering four cognitive domains—Understanding, Recall, Application, and Analysis—based on Bloom’s taxonomy. Thirty independent simulation runs were conducted with systematic variation of the model’s temperature parameter to gauge the balance between deterministic and creative responses. The generated responses underwent qualitative error analysis using a refined taxonomy that categorized errors such as “Unsupported Medical Claim,” “Hallucination of Information,” “Sticking with Wrong Diagnosis,” “Non-medical Factual Error,” “Incorrect Understanding of Task,” “Reasonable Response,” “Ignore Missing Information,” and “Incorrect or Vague Conclusion.” Two board-certified anesthesiologists performed independent annotations, with disagreements resolved by a third expert. Statistical evaluations—including one-way ANOVA, non-parametric tests, chi-square, and linear mixed-effects modeling—were used to compare performance across domains and analyze error frequency. RESULTS: GPT-4o achieved an overall accuracy of 83.69%. Performance varied significantly by cognitive domain, with the highest accuracy observed in the Understanding (90.10%) and Recall (84.38%) domains, and lower accuracy in Application (76.83%) and Analysis (76.54%). Among the 120 incorrect responses, unsupported medical claims were the most common error (40.69%), followed by vague or incorrect conclusions (22.07%). Co-occurrence analyses revealed that unsupported claims often appeared alongside imprecise conclusions, highlighting a trend of compounded errors particularly in tasks requiring complex reasoning. Inter-rater reliability for error annotation was robust, with a mean Cohen’s kappa of 0.73. CONCLUSIONS: While GPT-4o exhibits strengths in factual recall and comprehension, its limitations in handling higher-order reasoning and diagnostic judgment are evident through frequent unsupported medical claims and vague conclusions. These findings underscore the need for improved domain-specific fine-tuning, enhanced error mitigation strategies, and integrated knowledge verification mechanisms prior to clinical deployment.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

AnesthesiologyEducational MeasurementClinical CompetenceLarge Language ModelsChileHumansGenerative Artificial IntelligenceReproducibility of Results

Stichwörter der Autor:innen

AnesthesiologyClinical decision supportDiagnostic reasoning.GPT-4oLarge language modelsMedical AI evaluationSpanish-language exam

Autor:innen und Institutionen

  • Division of Anesthesiology, School of Medicine, Pontificia Universidad Católica de Chile, Marcoleta 377, Santiago, Chile. falterma@uc.cl.
  • Department of Computer Science, School of Engineering, Pontificia Universidad Católica de Chile, Vicuña Mackenna 6840, Santiago, Chile.
  • National Center for Artificial Intelligence (CENIA), National Research and Development Agency (ANID), Vicuña Mackenna 6840, Santiago, Chile.
  • Health Science Department, School of Medicine, Pontificia Universidad Católica de Chile, Vicuña Mackenna 6840, Santiago, Chile.
  • Millennium Institute for Foundational Research on Data (IMFD), Vicuña Mackenna 6840, Santiago, Chile.
  • Division of Anesthesiology, School of Medicine, Pontificia Universidad Católica de Chile, Marcoleta 377, Santiago, Chile.

Förderung

  • Agencia Nacional de Investigación y Desarrollo
  • National Center for Artificial Intelligence

Zitieren

APA

Altermatt, F. R., Neyem, A., Sumonte, N. I., Villagrán, I., Mendoza, M., Lacassie, H. J., & Delfino, A. E. (2025). Evaluating GPT-4o in high-stakes medical assessments: performance and error analysis on a Chilean anesthesiology exam. BMC Medical Education, 25(1). https://doi.org/10.1186/s12909-025-08084-9

Vancouver

Altermatt FR, Neyem A, Sumonte NI, Villagrán I, Mendoza M, Lacassie HJ, et al. Evaluating GPT-4o in high-stakes medical assessments: performance and error analysis on a Chilean anesthesiology exam. BMC Medical Education. 2025;25(1). doi:10.1186/s12909-025-08084-9

BibTeX
@article{Altermatt2025,
  author = {Altermatt, Fernando R and Neyem, Andres and Sumonte, Nicolás I and Villagrán, Ignacio and Mendoza, Marcelo and Lacassie, Hector J and Delfino, Alejandro E},
  title = {Evaluating GPT-4o in high-stakes medical assessments: performance and error analysis on a Chilean anesthesiology exam},
  journal = {BMC Medical Education},
  year = {2025},
  volume = {25},
  number = {1},
  doi = {10.1186/s12909-025-08084-9}
}
Kostenlos starten →