EUVIMEDSkopri EUVIMED
LeitlinieQuerschnittstudieOpen access

Evaluating Large Language Models and Retrieval-Augmented Generation Enhancement for Delivering Guideline-Adherent Nutrition Information for Cardiovascular Disease Prevention: Cross-Sectional Study

Vijaya Parameswaran iD, Jenna Bernard iD, Alec Bernard iD, Neil Deo iD, Sean Tsung iD, Kalle Lyytinen iD, Christopher Sharp iD, Fatima Rodriguez iD, David J Maron iD, Rajesh Dash iD

Journal of Medical Internet Research · 2025-10-07 · 27 · e78625

DOI: 10.2196/78625 ↗

At a glance

Leitlinie in Journal of Medical Internet Research, published online on 2025-10-07. Written by Vijaya Parameswaran and 9 co-authors. Institution: School of Medicine, Stanford University, Palo Alto, CA, United States. Major MEDLINE topics: Cardiovascular Diseases, Language, Guideline Adherence. Cited 24 times so far (Crossref); 36 references. The full text is freely available in PubMed Central. Funded by: NHLBI NIH HHS.

AstrattSubjectsDettalji biblijografiċiCiteRelated studies

Abstract (original language)

Background

Cardiovascular disease (CVD) remains the leading cause of death worldwide, yet many web-based sources on cardiovascular (CV) health are inaccessible. Large language models (LLMs) are increasingly used for health-related inquiries and offer an opportunity to produce accessible and scalable CV health information. However, because these models are trained on heterogeneous data, including unverified user-generated content, the quality and reliability of food and nutrition information on CVD prevention remain uncertain. Recent studies have examined LLM use in various health care applications, but their effectiveness for providing nutrition information remains understudied. Although retrieval-augmented generation (RAG) frameworks have been shown to enhance LLM consistency and accuracy, their use in delivering nutrition information for CVD prevention requires further evaluation.

Objective

To evaluate the effectiveness of off-the-shelf and RAG-enhanced LLMs in delivering guideline-adherent nutrition information for CVD prevention, we assessed 3 off-the-shelf models (ChatGPT-4o, Perplexity, and Llama 3-70B) and a Llama 3-70B+RAG model.

Methods

We curated 30 nutrition questions that comprehensively addressed CVD prevention. These were approved by a registered dietitian providing preventive cardiology services at an academic medical center and were posed 3 times to each model. We developed a 15,074-word knowledge bank incorporating the American Heart Association’s 2021 dietary guidelines and related website content to enhance Meta’s Llama 3-70B model using RAG. The model received this and a few-shot prompt as context, included citations in a Context Source section, and used vector similarity to align responses with guideline content, with the temperature parameter set to 0.5 to enhance consistency. Model responses were evaluated by 3 expert reviewers against benchmark CV guidelines for appropriateness, reliability, readability, harm, and guideline adherence. Mean scores were compared using ANOVA, with statistical significance set at P70%; P

Subjects

MeSH terms (MEDLINE)

Cardiovascular DiseasesLanguageGuideline AdherenceHumansCross-Sectional StudiesInternetLarge Language Models

Author keywords

artificial intelligencecardiovascular dietary guidelineslarge language modelsqualitative evaluationretrieval-augmented generation

Authors and institutions

  • School of Medicine, Stanford University, Palo Alto, CA, United States.
  • Tufts University, Boston, MA, United States.
  • University of Illinois Urbana-Champaign, Champaign, IL, United States.
  • Stanford University, Palo Alto, CA, United States.
  • Case Western Reserve University, Cleveland, OH, United States.

ORCID profiles: Vijaya Parameswaran, Jenna Bernard, Alec Bernard, Neil Deo, Sean Tsung, Kalle Lyytinen, Christopher Sharp, Fatima Rodriguez, David J Maron, Rajesh Dash

Funding

  • NHLBI NIH HHS

Cite

APA

Parameswaran, V., Bernard, J., Bernard, A., Deo, N., Tsung, S., Lyytinen, K., Sharp, C., Rodriguez, F., Maron, D. J., & Dash, R. (2025). Evaluating Large Language Models and Retrieval-Augmented Generation Enhancement for Delivering Guideline-Adherent Nutrition Information for Cardiovascular Disease Prevention: Cross-Sectional Study. Journal of Medical Internet Research, 27, e78625. https://doi.org/10.2196/78625

Vancouver

Parameswaran V, Bernard J, Bernard A, Deo N, Tsung S, Lyytinen K, et al. Evaluating Large Language Models and Retrieval-Augmented Generation Enhancement for Delivering Guideline-Adherent Nutrition Information for Cardiovascular Disease Prevention: Cross-Sectional Study. Journal of Medical Internet Research. 2025;27:e78625. doi:10.2196/78625

BibTeX
@article{Parameswaran2025,
  author = {Parameswaran, Vijaya and Bernard, Jenna and Bernard, Alec and Deo, Neil and Tsung, Sean and Lyytinen, Kalle and Sharp, Christopher and Rodriguez, Fatima and Maron, David J and Dash, Rajesh},
  title = {Evaluating Large Language Models and Retrieval-Augmented Generation Enhancement for Delivering Guideline-Adherent Nutrition Information for Cardiovascular Disease Prevention: Cross-Sectional Study},
  journal = {Journal of Medical Internet Research},
  year = {2025},
  volume = {27},
  pages = {e78625},
  doi = {10.2196/78625}
}
Start for free →