EUVIMEDOpdag EUVIMED
Open access

Retrieval-Augmented Generation for Medical Question Answering on a Heart Failure Dataset: Performance Analysis

Shiran Zhang iD, Evelyn Phan iD, Pedro Velmovitsky iD, Quynh Pham iD, Scott Sanner iD

JMIR Formative Research · 2026

DOI: 10.2196/84932 ↗

At a glance

Journal article in JMIR Formative Research, 2026. Written by Shiran Zhang and 4 co-authors. Institution: Department of Mechanical & Industrial Engineering, Faculty of Applied Science & Engineering, University of Toronto, 27 King's College Circle, Toronto…. Major MEDLINE topics: Heart Failure, Information Storage and Retrieval. Cited 5 times so far (Crossref); 47 references. The full text is freely available in PubMed Central.

AbstractSubjectsBibliografiske oplysningerCiteRelated studies

Abstract (original language)

Background

The integration of retrieval-augmented generation (RAG) systems into the domain of medical question-answering (QA) presents a significant opportunity to enhance the effectiveness and accuracy of clinical support systems.

Objective

This study aimed to explore the design choices within the RAG framework and the use of large language model (LLM) classifiers to optimize medical QA systems, enhancing response quality for patient and caregiver queries of varying risk levels.

Methods

In total, we curated a dataset of 109 patient and caregiver questions related to heart failure (HF)—categorized into answerable (direct, fact-based queries), helpful deferral (general guidance or lifestyle advisory queries), and nonanswerable (out-of-scope or high-risk and medical intervention queries) types—along with relevant documents and a target answer for each question from the website The Heart Hub . Applying a system architecture leveraging RAG with a structured query taxonomy and robust classification mechanisms, this paper provided an empirical assessment for medical QA on a HF dataset and introduced a QA system pipeline design, providing a foundation for extended application across various medical fields. Specifically, we evaluated design choices in the initial retrieval stage of RAG and their impact on performance. We assessed final answer quality from the generation stage using popular passage scoring methods for QA, such as Recall-Oriented Understudy for Gisting Evaluation (ROUGE), BERTScore, and Intersection over Union score.

Results

The pipeline first uses an LLM-based classifier, achieving 65% accuracy for answerable and helpful deferral queries and 100% accuracy for identifying nonanswerable queries. In information retrieval, the BioMedical Contrastive Pre-trained Transformers (MedCPT) cross encoder performed best as a dense retrieval method, delivering an average of 93% recall @ 7 through ranked relevance scores to obtain the top documents with recall @ k denoting recall computed over the top-k retrieved items. For further retrieving snippets from such documents, its average performance was 72.5% for sentence-level snippets and 83% for paragraph-level snippets. A second LLM-based classifier, used to refine the generated responses, resulted in an overall reduction in ROUGE-1 recall by 13% and Bidirectional Encoder Representations from Transformers (BERT) precision by 11%. However, Intersection over Union scores, or the overlap between “gold answers” and system answers, increased by 24%, demonstrating enhanced alignment with ground truth responses. This also indicates the system’s improved ability to generate concise and accurate medical responses.

Conclusions

The implementation of a structured RAG framework paired with LLM classifiers for medical QA introduces a promising avenue for enhancing clinical decision support systems. By systematically analyzing the impact of query taxonomy, retrieval configurations, and response strategies, this approach clarifies the relative importance of each component within the medical RAG system using a HF dataset. Our findings provide actionable guidance on optimal design choices for maximizing retrieval and response accuracy; thus, informing the development of robust, scalable medical QA systems.

Subjects

MeSH terms (MEDLINE)

Heart FailureInformation Storage and RetrievalHumansLarge Language Models

Author keywords

healthcare technologyheart failureinformation retrievalmachine learningmedical question-answeringretrieval-augmented generation

Authors and institutions

  • Department of Mechanical & Industrial Engineering, Faculty of Applied Science & Engineering, University of Toronto, 27 King's College Circle, Toronto, ON, M5S 1A1, Canada, 1 416-978-2011.
  • The Edward S. Rogers Sr. Department of Electrical and Computer Engineering, Faculty of Applied Science & Engineering, University of Toronto, Toronto, ON, Canada.
  • Centre for Digital Therapeutics, Toronto General Hospital Research Institute, University Health Network, Toronto, ON, Canada.
  • Institute of Health Policy, Management, and Evaluation, Dalla Lana School of Public Health, University of Toronto, Toronto, ON, Canada.
  • Telfer School of Management, University of Ottawa, Ottawa, ON, Canada.
  • School of Public Health Sciences, Faculty of Health, University of Waterloo, Waterloo, ON, Canada.

ORCID profiles: Shiran Zhang, Evelyn Phan, Pedro Velmovitsky, Quynh Pham, Scott Sanner

Cite

APA

Zhang, S., Phan, E., Velmovitsky, P., Pham, Q., & Sanner, S. (2026). Retrieval-Augmented Generation for Medical Question Answering on a Heart Failure Dataset: Performance Analysis. JMIR Formative Research. https://doi.org/10.2196/84932

Vancouver

Zhang S, Phan E, Velmovitsky P, Pham Q, Sanner S. Retrieval-Augmented Generation for Medical Question Answering on a Heart Failure Dataset: Performance Analysis. JMIR Formative Research. 2026. doi:10.2196/84932

BibTeX
@article{Zhang2026,
  author = {Zhang, Shiran and Phan, Evelyn and Velmovitsky, Pedro and Pham, Quynh and Sanner, Scott},
  title = {Retrieval-Augmented Generation for Medical Question Answering on a Heart Failure Dataset: Performance Analysis},
  journal = {JMIR Formative Research},
  year = {2026},
  doi = {10.2196/84932}
}
Start for free →