EUVIMEDEUVIMED entdecken
VergleichsstudieLeitlinie

Performance of Large Language Models on the Acute Coronary Syndrome Guidelines Using Retrieval-Augmented Generation

Michaella Alexandrou, Sant Kumar, Arun Umesh Mahtani, Dimitrios Strepkos, Pedro E P Carvalho, Deniz Mutlu, Ozgur Selim Ser, Athanasios Rempakos, Olga C Mastrodemos, Bavana V Rangan, Sandeep Jalli, Yader Sandoval und 1 weitere

JACC: Cardiovascular Interventions · 2025

DOI: 10.1016/j.jcin.2025.08.019 ↗

Kurz gefasst

Vergleichsstudie in der Zeitschrift JACC: Cardiovascular Interventions, 2025. Verfasst von Michaella Alexandrou und 12 weiteren Autor:innen. Beteiligte Einrichtung: Minneapolis Heart Institute and Minneapolis Heart Institute Foundation, Abbott Northwestern Hospital, Minneapolis, Minnesota, USA. Hauptthemen laut MEDLINE: Acute Coronary Syndrome, Practice Guidelines as Topic, Decision Support Techniques, Natural Language Processing. Die Arbeit wurde bisher 11-mal zitiert (Crossref) und nennt 42 Literaturangaben. Gefördert von: Minneapolis Heart Institute Foundation, Joseph Durda Foundation.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

BACKGROUND: Large language models (LLMs) are increasingly applied in interventional cardiology, but hallucinations limit their clinical utility. OBJECTIVES: The aim of this study was to assess whether retrieval-augmented generation (RAG), a technique that allows LLMs to access guideline content during response generation, improves accuracy when answering questions on the basis of the guidelines for acute coronary syndromes. METHODS: The accuracy of ChatGPT-4o, DeepSeek R1, and Med-PaLM 2 was compared using a set of 38 open-ended cardiology guideline-based questions and answers. ChatGPT-4o and DeepSeek R1 were evaluated with and without RAG, while Med-PaLM 2, a medicine-specific LLM, was tested without RAG. Model outputs were compared against guideline recommendations using an artificial intelligence-powered similarity score tool. RESULTS: DeepSeek R1 with RAG achieved the highest accuracy (94.7%; 95% CI: 82.7%-98.5%), followed by ChatGPT-4o with RAG (92.1%; 95% CI: 79.2%-97.3%) (P = 0.922). ChatGPT-4o without RAG achieved 71.1% accuracy (95% CI: 55.2%-83.0%), which significantly improved with RAG (P = 0.017). Among non-RAG models, DeepSeek R1 demonstrated the highest accuracy (78.9%; 95% CI: 63.7%-88.9%), followed by ChatGPT-4o without RAG (71.1%) (P = 0.083). Med-PaLM 2 had the lowest accuracy (68.4%; 95% CI: 52.5%-80.9%). Spearman correlation analysis revealed a strong correlation between DeepSeek R1 without RAG and Med-PaLM 2 (r = 0.646; 95% CI: 0.411-0.800; P < 0.001), indicating similar response patterns. Scatterplot analysis further revealed that RAG disproportionately improved lower scoring questions in DeepSeek R1 while improving scores more evenly in ChatGPT-4o. CONCLUSIONS: Embedding guideline content into LLM workflows via RAG can enhance LLM accuracy for clinical applications, particularly in scenarios common to interventional cardiology. These results support the potential for LLMs, when enhanced with domain-specific knowledge, to optimize clinical decision making and increase alignment with guidelines.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Acute Coronary SyndromePractice Guidelines as TopicDecision Support TechniquesNatural Language ProcessingDecision Support Systems, ClinicalHumansReproducibility of ResultsClinical Decision-MakingLarge Language Models

Stichwörter der Autor:innen

artificial intelligencecardiologyguidelineslarge language model

Autor:innen und Institutionen

  • Minneapolis Heart Institute and Minneapolis Heart Institute Foundation, Abbott Northwestern Hospital, Minneapolis, Minnesota, USA.
  • Department of Cardiology, Creighton University School of Medicine, Phoenix, Arizona, USA.
  • Department of Cardiology, Virginia Commonwealth University, Richmond, Virginia, USA.
  • Department of Internal Medicine, Corewell Health William Beaumont University Hospital, Royal Oak, Michigan, USA.
  • Minneapolis Heart Institute and Minneapolis Heart Institute Foundation, Abbott Northwestern Hospital, Minneapolis, Minnesota, USA. Electronic address: esbrilakis@gmail.com.

Förderung

  • Minneapolis Heart Institute Foundation
  • Joseph Durda Foundation

Zitieren

APA

Alexandrou, M., Kumar, S., Mahtani, A. U., Strepkos, D., Carvalho, P. E. P., Mutlu, D., Ser, O. S., Rempakos, A., Mastrodemos, O. C., Rangan, B. V., Jalli, S., Sandoval, Y., & Brilakis, E. S. (2025). Performance of Large Language Models on the Acute Coronary Syndrome Guidelines Using Retrieval-Augmented Generation. JACC: Cardiovascular Interventions. https://doi.org/10.1016/j.jcin.2025.08.019

Vancouver

Alexandrou M, Kumar S, Mahtani AU, Strepkos D, Carvalho PEP, Mutlu D, et al. Performance of Large Language Models on the Acute Coronary Syndrome Guidelines Using Retrieval-Augmented Generation. JACC: Cardiovascular Interventions. 2025. doi:10.1016/j.jcin.2025.08.019

BibTeX
@article{Alexandrou2025,
  author = {Alexandrou, Michaella and Kumar, Sant and Mahtani, Arun Umesh and Strepkos, Dimitrios and Carvalho, Pedro E P and Mutlu, Deniz and Ser, Ozgur Selim and Rempakos, Athanasios and Mastrodemos, Olga C and Rangan, Bavana V and Jalli, Sandeep and Sandoval, Yader and Brilakis, Emmanouil S},
  title = {Performance of Large Language Models on the Acute Coronary Syndrome Guidelines Using Retrieval-Augmented Generation},
  journal = {JACC: Cardiovascular Interventions},
  year = {2025},
  doi = {10.1016/j.jcin.2025.08.019}
}
Kostenlos starten →