EUVIMEDEUVIMED entdecken
Übersichtsarbeit

Evaluating Large Language Models on American Board of Anesthesiology-style Anesthesiology Questions: Accuracy, Domain Consistency, and Clinical Implications

Sagar Patel, Vinh Ngo, Brian Wilhelmi

Journal of Cardiothoracic and Vascular Anesthesia · 2025

DOI: 10.1053/j.jvca.2025.05.033 ↗

Kurz gefasst

Übersichtsarbeit in der Zeitschrift Journal of Cardiothoracic and Vascular Anesthesia, 2025. Verfasst von Sagar Patel und 2 weiteren Autor:innen. Beteiligte Einrichtung: Creighton University School of Medicine, Phoenix, AZ. Electronic address: sagarpatel@creighton.edu. Hauptthemen laut MEDLINE: Anesthesiology, Educational Measurement, Specialty Boards, Language. Die Arbeit wurde bisher 11-mal zitiert (Crossref) und nennt 9 Literaturangaben.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Recent advances in large language models (LLMs) have led to growing interest in their potential applications in medical education and clinical practice. This study evaluated whether five widely used and highly developed LLMs-ChatGPT-4, Gemini, Claude, Microsoft CoPilot, and Meta-could achieve a passing score on the American Board of Anesthesiology (ABA) BASIC Exam. Each model completed three separate sets of 200 multiple-choice questions derived from a widely used review resource, with the content distribution mirroring the ABA BASIC Exam blueprint. All five models demonstrated statistically significant performance above the 70% passing threshold (p < 0.05), with the following averages: ChatGPT-4: 92.0%, Gemini: 89.0%, Claude: 88.3%, Microsoft CoPilot: 91.5%, and Meta: 85.8%. Furthermore, an analysis of variance comparing their mean accuracy scores found no statistically significant difference among them (F = 1.88, p = 0.190). These findings suggest that current LLMs can surpass the minimum competency required for board certification, raising important questions about their future role in medical education and clinical decision-making. Performance on topics central to cardiac, thoracic, and vascular anesthesiology-such as hemodynamic management, cardiopulmonary physiology, and coagulation-was particularly robust, suggesting relevance to both fellowship-level education and complex intraoperative care. While these results highlight the capability of artificial intelligence (AI) to meet standardized medical knowledge benchmarks, their broader implications extend beyond examination performance. As AI continues to evolve, its integration into real-time patient care may transform anesthesiology practice, offering decision-support tools that assist physicians in synthesizing complex clinical data. Further research is needed to explore the reliability, ethical considerations, and real-world applications of AI-driven technologies in patient care settings.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

AnesthesiologyEducational MeasurementSpecialty BoardsLanguageClinical CompetenceHumansUnited StatesLarge Language Models

Stichwörter der Autor:innen

BASIC Examanesthesiologyartificial intelligencebenchmarkingclinical decision supportlarge language modelsmedical educationnatural language processing

Autor:innen und Institutionen

  • Creighton University School of Medicine, Phoenix, AZ. Electronic address: sagarpatel@creighton.edu.
  • Creighton University School of Medicine, Phoenix, AZ.
  • Creighton University School of Medicine, Phoenix, AZ; Barrow Neurological Institute, Department of Anesthesiology, Phoenix, AZ.

Zitieren

APA

Patel, S., Ngo, V., & Wilhelmi, B. (2025). Evaluating Large Language Models on American Board of Anesthesiology-style Anesthesiology Questions: Accuracy, Domain Consistency, and Clinical Implications. Journal of Cardiothoracic and Vascular Anesthesia. https://doi.org/10.1053/j.jvca.2025.05.033

Vancouver

Patel S, Ngo V, Wilhelmi B. Evaluating Large Language Models on American Board of Anesthesiology-style Anesthesiology Questions: Accuracy, Domain Consistency, and Clinical Implications. Journal of Cardiothoracic and Vascular Anesthesia. 2025. doi:10.1053/j.jvca.2025.05.033

BibTeX
@article{Patel2025,
  author = {Patel, Sagar and Ngo, Vinh and Wilhelmi, Brian},
  title = {Evaluating Large Language Models on American Board of Anesthesiology-style Anesthesiology Questions: Accuracy, Domain Consistency, and Clinical Implications},
  journal = {Journal of Cardiothoracic and Vascular Anesthesia},
  year = {2025},
  doi = {10.1053/j.jvca.2025.05.033}
}
Kostenlos starten →