Evaluating the Performance of Large Language Models in Vascular Surgery Cases: Cross-Sectional Study
JMIR Perioperative Medicine · 2026-11-06 · 9 · e91973-e91973
DOI: 10.2196/91973 ↗At a glance
Querschnittstudie in JMIR Perioperative Medicine, published online on 2026-11-06. Written by Asanka Wijetunga and 3 co-authors. Institution: Westmead Hospital, Cnr Darcy and Hawkesbury Road, Westmead, New South Wales, 2145, Australia, 61 8890 5555 ext 9. Major MEDLINE topics: Large Language Models, Vascular Surgical Procedures. 35 references. The full text is freely available in PubMed Central.
Abstract (original language)
Background
Large language models (LLMs) such as ChatGPT, Gemini, and Claude are increasingly used by clinicians, yet their accuracy, safety, and consistency in clinical cases remain poorly defined. Most studies assess LLMs using multiple-choice questions rather than free-response reasoning.
Objective
The primary objective of this study was to evaluate the performance and safety of 3 widely used LLMs in realistic vascular surgery scenarios in key domains such as diagnosis, investigation, and multidisciplinary and multimodal management. The secondary objective of this study was to evaluate the safety of the models’ outputs.
Methods
A total of 42 fictitious vascular cases across 5 major pathologies (acute limb ischemia [ALI], aortopathy, chronic limb ischemia, diabetic foot infection, and extracranial cerebrovascular disease) were independently entered into GPT-5, Gemini 2.5, and Claude Sonnet 4.5 using standardized prompts. Each model answered structured questions covering diagnosis, investigation, and management (both operative and nonoperative). Responses were scored out of 20 by a panel using a predefined rubric. Each plan’s overall safety was separately assessed. Comparative analyses used 2-tailed t tests, ANOVA, and multivariable logistic regression.
Results
Mean composite scores by model were 86.5% for ChatGPT, 83.5% for Gemini, and 88.0% for Claude, whereas scores by disease were 82.7% for ALI, 83.8% for aortopathy, 85.9% for chronic limb ischemia, 86.1% for diabetic foot infection, and 80.7% for extracranial cerebrovascular disease ( P >.05 in each case). Unsafe plans were generated in 11.9%, 23.8%, and 7.1% of cases by ChatGPT, Gemini, and Claude, respectively. On multivariable analysis, independent predictors of unsafe outputs were lower composite score (odds ratio [OR] 0.471, 95% CI 0.306‐0.722; P =.001), higher word count (OR 1.003, 95% CI 1.001‐1.005; P =.001), and ALI (OR 20.9, 95% CI 3.79-61.8; P
Subjects
MeSH terms (MEDLINE)
Author keywords
Authors and institutions
- Westmead Hospital, Cnr Darcy and Hawkesbury Road, Westmead, New South Wales, 2145, Australia, 61 8890 5555 ext 9.
- The University of Sydney, Sydney, New South Wales, Australia.
Cite
APA
Wijetunga, A., Wang, Y., Yaghi, C., & Vicaretti, M. (2026). Evaluating the Performance of Large Language Models in Vascular Surgery Cases: Cross-Sectional Study. JMIR Perioperative Medicine, 9, e91973-e91973. https://doi.org/10.2196/91973
Vancouver
Wijetunga A, Wang Y, Yaghi C, Vicaretti M. Evaluating the Performance of Large Language Models in Vascular Surgery Cases: Cross-Sectional Study. JMIR Perioperative Medicine. 2026;9:e91973-e91973. doi:10.2196/91973
BibTeX
@article{Wijetunga2026,
author = {Wijetunga, Asanka and Wang, Yunyi and Yaghi, Chantell and Vicaretti, Mauro},
title = {Evaluating the Performance of Large Language Models in Vascular Surgery Cases: Cross-Sectional Study},
journal = {JMIR Perioperative Medicine},
year = {2026},
volume = {9},
pages = {e91973--e91973},
doi = {10.2196/91973}
}
EUVIMED