Performance of Large Language Models on the Acute Coronary Syndrome Guidelines Using Retrieval-Augmented Generation
JACC: Cardiovascular Interventions · 2025
DOI: 10.1016/j.jcin.2025.08.019 ↗Kurz gefasst
Vergleichsstudie in der Zeitschrift JACC: Cardiovascular Interventions, 2025. Verfasst von Michaella Alexandrou und 12 weiteren Autor:innen. Beteiligte Einrichtung: Minneapolis Heart Institute and Minneapolis Heart Institute Foundation, Abbott Northwestern Hospital, Minneapolis, Minnesota, USA. Hauptthemen laut MEDLINE: Acute Coronary Syndrome, Practice Guidelines as Topic, Decision Support Techniques, Natural Language Processing. Die Arbeit wurde bisher 11-mal zitiert (Crossref) und nennt 42 Literaturangaben. Gefördert von: Minneapolis Heart Institute Foundation, Joseph Durda Foundation.
Abstract (Originalsprache)
BACKGROUND: Large language models (LLMs) are increasingly applied in interventional cardiology, but hallucinations limit their clinical utility. OBJECTIVES: The aim of this study was to assess whether retrieval-augmented generation (RAG), a technique that allows LLMs to access guideline content during response generation, improves accuracy when answering questions on the basis of the guidelines for acute coronary syndromes. METHODS: The accuracy of ChatGPT-4o, DeepSeek R1, and Med-PaLM 2 was compared using a set of 38 open-ended cardiology guideline-based questions and answers. ChatGPT-4o and DeepSeek R1 were evaluated with and without RAG, while Med-PaLM 2, a medicine-specific LLM, was tested without RAG. Model outputs were compared against guideline recommendations using an artificial intelligence-powered similarity score tool. RESULTS: DeepSeek R1 with RAG achieved the highest accuracy (94.7%; 95% CI: 82.7%-98.5%), followed by ChatGPT-4o with RAG (92.1%; 95% CI: 79.2%-97.3%) (P = 0.922). ChatGPT-4o without RAG achieved 71.1% accuracy (95% CI: 55.2%-83.0%), which significantly improved with RAG (P = 0.017). Among non-RAG models, DeepSeek R1 demonstrated the highest accuracy (78.9%; 95% CI: 63.7%-88.9%), followed by ChatGPT-4o without RAG (71.1%) (P = 0.083). Med-PaLM 2 had the lowest accuracy (68.4%; 95% CI: 52.5%-80.9%). Spearman correlation analysis revealed a strong correlation between DeepSeek R1 without RAG and Med-PaLM 2 (r = 0.646; 95% CI: 0.411-0.800; P < 0.001), indicating similar response patterns. Scatterplot analysis further revealed that RAG disproportionately improved lower scoring questions in DeepSeek R1 while improving scores more evenly in ChatGPT-4o. CONCLUSIONS: Embedding guideline content into LLM workflows via RAG can enhance LLM accuracy for clinical applications, particularly in scenarios common to interventional cardiology. These results support the potential for LLMs, when enhanced with domain-specific knowledge, to optimize clinical decision making and increase alignment with guidelines.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Minneapolis Heart Institute and Minneapolis Heart Institute Foundation, Abbott Northwestern Hospital, Minneapolis, Minnesota, USA.
- Department of Cardiology, Creighton University School of Medicine, Phoenix, Arizona, USA.
- Department of Cardiology, Virginia Commonwealth University, Richmond, Virginia, USA.
- Department of Internal Medicine, Corewell Health William Beaumont University Hospital, Royal Oak, Michigan, USA.
- Minneapolis Heart Institute and Minneapolis Heart Institute Foundation, Abbott Northwestern Hospital, Minneapolis, Minnesota, USA. Electronic address: esbrilakis@gmail.com.
Förderung
- Minneapolis Heart Institute Foundation
- Joseph Durda Foundation
Zitieren
APA
Alexandrou, M., Kumar, S., Mahtani, A. U., Strepkos, D., Carvalho, P. E. P., Mutlu, D., Ser, O. S., Rempakos, A., Mastrodemos, O. C., Rangan, B. V., Jalli, S., Sandoval, Y., & Brilakis, E. S. (2025). Performance of Large Language Models on the Acute Coronary Syndrome Guidelines Using Retrieval-Augmented Generation. JACC: Cardiovascular Interventions. https://doi.org/10.1016/j.jcin.2025.08.019
Vancouver
Alexandrou M, Kumar S, Mahtani AU, Strepkos D, Carvalho PEP, Mutlu D, et al. Performance of Large Language Models on the Acute Coronary Syndrome Guidelines Using Retrieval-Augmented Generation. JACC: Cardiovascular Interventions. 2025. doi:10.1016/j.jcin.2025.08.019
BibTeX
@article{Alexandrou2025,
author = {Alexandrou, Michaella and Kumar, Sant and Mahtani, Arun Umesh and Strepkos, Dimitrios and Carvalho, Pedro E P and Mutlu, Deniz and Ser, Ozgur Selim and Rempakos, Athanasios and Mastrodemos, Olga C and Rangan, Bavana V and Jalli, Sandeep and Sandoval, Yader and Brilakis, Emmanouil S},
title = {Performance of Large Language Models on the Acute Coronary Syndrome Guidelines Using Retrieval-Augmented Generation},
journal = {JACC: Cardiovascular Interventions},
year = {2025},
doi = {10.1016/j.jcin.2025.08.019}
}
EUVIMED