Cardiology knowledge assessment of retrieval-augmented open versus proprietary large language models
PLOS Digital Health · 12.03.2026 · 5 (3) · e0001029
DOI: 10.1371/journal.pdig.0001029 ↗Kurz gefasst
Fachartikel in der Zeitschrift PLOS Digital Health, online erschienen am 12.03.2026. Verfasst von Constantine Tarabanis und 11 weiteren Autor:innen. Beteiligte Einrichtung: Cardiology Division, Heart and Vascular Institute, Mass General Brigham, Boston, Massachusetts, United States of America. Die Arbeit wurde bisher 5-mal zitiert (Crossref) und nennt 21 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: NHLBI NIH HHS.
Abstract (Originalsprache)
To evaluate the performance of open-weight and proprietary LLMs, with and without Retrieval-Augmented Generation (RAG), on cardiology board-style questions and benchmark them against the human average. We tested 14 LLMs (6 open-weight, 8 proprietary) on 449 multiple-choice questions from the American College of Cardiology Self-Assessment Program (ACCSAP). Accuracy was measured as percent correct. RAG was implemented using a knowledge base of 123 guideline and textbook documents. The open-weight model DeepSeek R1 achieved the highest accuracy at 86.9% (95% CI: 83.4–89.7%), outperforming proprietary models and the human average of 78%. GPT 4o (80.9%, 95% CI: 77.0–84.2%) and the commercial platform OpenEvidence (81.3%, 95% CI: 77.4–84.7%) demonstrated similar performance. A positive correlation between model size and performance was observed within model families, but across families, substantial variability persisted among models with similar parameter counts. After RAG, all models improved, and open-weight models like Mistral Large 2 (78.0%, 95% CI: 73.9–81.5) performed comparably to proprietary alternatives like GPT 4o. Large language models (LLMs) are increasingly integrated into clinical workflows, yet their performance in cardiovascular medicine remains insufficiently evaluated. Open-weight models can match or exceed proprietary systems in cardiovascular knowledge, with RAG particularly beneficial for smaller models. Given their transparency, configurability, and potential for local deployment, open-weight models, strategically augmented, represent viable, lower-cost alternatives for clinical applications. Open-weight LLMs demonstrate competency in cardiovascular medicine comparable to or exceeding that of proprietary models, with and without RAG depending on the model.
Autor:innen und Institutionen
- Cardiology Division, Heart and Vascular Institute, Mass General Brigham, Boston, Massachusetts, United States of America.
- Cardiovascular Disease Initiative, Broad Institute of MIT and Harvard, Cambridge, Massachusetts, United States of America.
- Telemachus and Irene Demoulas Family Foundation Center for Cardiac Arrhythmias, Massachusetts General Hospital, Boston, Massachusetts, United States of America.
- Cardiovascular Research Center, Massachusetts General Hospital, Boston, Massachusetts, United States of America.
- Information Systems Laboratory, University of Macedonia, Thessaloniki, Greece.
- New York University School of Medicine, New York City, New York, United States of America.
ORCID-Profile: Constantine Tarabanis
Förderung
- NHLBI NIH HHS
Zitieren
APA
Tarabanis, C., Khurshid, S., Karamanou, A., Piperaki, R., Mavromatis, L. A., Hatzimemos, A., Tachmatzidis, D., Bakogiannis, C., Vassilikos, V., Ellinor, P. T., Jankelson, L., & Kalampokis, E. (2026). Cardiology knowledge assessment of retrieval-augmented open versus proprietary large language models. PLOS Digital Health, 5(3), e0001029. https://doi.org/10.1371/journal.pdig.0001029
Vancouver
Tarabanis C, Khurshid S, Karamanou A, Piperaki R, Mavromatis LA, Hatzimemos A, et al. Cardiology knowledge assessment of retrieval-augmented open versus proprietary large language models. PLOS Digital Health. 2026;5(3):e0001029. doi:10.1371/journal.pdig.0001029
BibTeX
@article{Tarabanis2026,
author = {Tarabanis, Constantine and Khurshid, Shaan and Karamanou, Areti and Piperaki, Rodo and Mavromatis, Lucas A and Hatzimemos, Aris and Tachmatzidis, Dimitrios and Bakogiannis, Constantinos and Vassilikos, Vassilios and Ellinor, Patrick T and Jankelson, Lior and Kalampokis, Evangelos},
title = {Cardiology knowledge assessment of retrieval-augmented open versus proprietary large language models},
journal = {PLOS Digital Health},
year = {2026},
volume = {5},
number = {3},
pages = {e0001029},
doi = {10.1371/journal.pdig.0001029}
}
EUVIMED