Evaluating the Performance of Large Language Models in Vascular Surgery Cases: Cross-Sectional Study
JMIR Perioperative Medicine · 06.11.2026 · 9 · e91973-e91973
DOI: 10.2196/91973 ↗Kurz gefasst
Querschnittstudie in der Zeitschrift JMIR Perioperative Medicine, online erschienen am 06.11.2026. Verfasst von Asanka Wijetunga und 3 weiteren Autor:innen. Beteiligte Einrichtung: Westmead Hospital, Cnr Darcy and Hawkesbury Road, Westmead, New South Wales, 2145, Australia, 61 8890 5555 ext 9. Hauptthemen laut MEDLINE: Large Language Models, Vascular Surgical Procedures. Die Arbeit nennt 35 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.
Abstract (Originalsprache)
Background
Large language models (LLMs) such as ChatGPT, Gemini, and Claude are increasingly used by clinicians, yet their accuracy, safety, and consistency in clinical cases remain poorly defined. Most studies assess LLMs using multiple-choice questions rather than free-response reasoning.
Objective
The primary objective of this study was to evaluate the performance and safety of 3 widely used LLMs in realistic vascular surgery scenarios in key domains such as diagnosis, investigation, and multidisciplinary and multimodal management. The secondary objective of this study was to evaluate the safety of the models’ outputs.
Methods
A total of 42 fictitious vascular cases across 5 major pathologies (acute limb ischemia [ALI], aortopathy, chronic limb ischemia, diabetic foot infection, and extracranial cerebrovascular disease) were independently entered into GPT-5, Gemini 2.5, and Claude Sonnet 4.5 using standardized prompts. Each model answered structured questions covering diagnosis, investigation, and management (both operative and nonoperative). Responses were scored out of 20 by a panel using a predefined rubric. Each plan’s overall safety was separately assessed. Comparative analyses used 2-tailed t tests, ANOVA, and multivariable logistic regression.
Results
Mean composite scores by model were 86.5% for ChatGPT, 83.5% for Gemini, and 88.0% for Claude, whereas scores by disease were 82.7% for ALI, 83.8% for aortopathy, 85.9% for chronic limb ischemia, 86.1% for diabetic foot infection, and 80.7% for extracranial cerebrovascular disease ( P >.05 in each case). Unsafe plans were generated in 11.9%, 23.8%, and 7.1% of cases by ChatGPT, Gemini, and Claude, respectively. On multivariable analysis, independent predictors of unsafe outputs were lower composite score (odds ratio [OR] 0.471, 95% CI 0.306‐0.722; P =.001), higher word count (OR 1.003, 95% CI 1.001‐1.005; P =.001), and ALI (OR 20.9, 95% CI 3.79-61.8; P
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Westmead Hospital, Cnr Darcy and Hawkesbury Road, Westmead, New South Wales, 2145, Australia, 61 8890 5555 ext 9.
- The University of Sydney, Sydney, New South Wales, Australia.
Zitieren
APA
Wijetunga, A., Wang, Y., Yaghi, C., & Vicaretti, M. (2026). Evaluating the Performance of Large Language Models in Vascular Surgery Cases: Cross-Sectional Study. JMIR Perioperative Medicine, 9, e91973-e91973. https://doi.org/10.2196/91973
Vancouver
Wijetunga A, Wang Y, Yaghi C, Vicaretti M. Evaluating the Performance of Large Language Models in Vascular Surgery Cases: Cross-Sectional Study. JMIR Perioperative Medicine. 2026;9:e91973-e91973. doi:10.2196/91973
BibTeX
@article{Wijetunga2026,
author = {Wijetunga, Asanka and Wang, Yunyi and Yaghi, Chantell and Vicaretti, Mauro},
title = {Evaluating the Performance of Large Language Models in Vascular Surgery Cases: Cross-Sectional Study},
journal = {JMIR Perioperative Medicine},
year = {2026},
volume = {9},
pages = {e91973--e91973},
doi = {10.2196/91973}
}
EUVIMED