EUVIMEDEUVIMED entdecken
LeitlinieOpen Access · CC BY-NC 4.0

Evaluating Adherence to Canadian Radiology Guidelines for Incidental Hepatobiliary Findings Using RAG-Enabled LLMs

Nicholas Dietrich iD, Brett Stubbert

Canadian Association of Radiologists Journal · 2025-11 · 76 (4) · 674-682

DOI: 10.1177/08465371251323124 ↗

Kurz gefasst

Leitlinie in der Zeitschrift Canadian Association of Radiologists Journal, online erschienen am 27.02.2025. Verfasst von Nicholas Dietrich und 1 weiteren Autor:innen. Beteiligte Einrichtung: Temerty Faculty of Medicine, University of Toronto, Toronto, ON, Canada. Hauptthemen laut MEDLINE: Guideline Adherence, Incidental Findings, Practice Guidelines as Topic, Liver Diseases. Die Arbeit wurde bisher 23-mal zitiert (Crossref) und nennt 34 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-NC 4.0).

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Purpose

Large language models (LLMs) have the potential to support clinical decision-making but often lack training on the latest clinical guidelines. Retrieval-augmented generation (RAG) may enhance guideline adherence by dynamically integrating external information. This study evaluates the performance of two LLMs, GPT-4o and o1-mini, with and without RAG, in adhering to Canadian radiology guidelines for incidental hepatobiliary findings.

Methods

A customized RAG architecture was developed to integrate guideline-based recommendations into LLM prompts. Clinical cases were curated and used to prompt models with and without RAG. Primary analyses assessed the rate of guideline adherence with comparisons made between LLMs with and without RAG. Secondary analyses evaluated reading ease, grade level, and response times for generated outputs.

Results

A total of 319 clinical cases were evaluated. Adherence rates were 81.7% for GPT-4o without RAG, 97.2% for GPT-4o with RAG, 79.3% for o1-mini without RAG, and 95.1% for o1-mini with RAG. Model performance differed significantly across groups, with RAG-enabled configurations outperforming their non-RAG counterparts ( P < .05). RAG-enabled models demonstrated improved reading ease and lower grade level scores; however, all model outputs remained at advanced comprehension levels. Response times for RAG-enabled models increased slightly due to additional retrieval processing but remained clinically acceptable.

Conclusions

RAG-enabled LLMs significantly improved adherence to Canadian radiology guidelines for incidental hepatobiliary findings without compromising readability or response times. This approach holds promise for advancing evidence-based care and warrants further validation across broader clinical settings.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Guideline AdherenceIncidental FindingsPractice Guidelines as TopicLiver DiseasesBiliary Tract DiseasesLanguageHumansCanadaFemaleMaleMiddle Aged

Stichwörter der Autor:innen

artificial intelligencehepatobiliaryimaging guidelinesincidental findingslarge language modelretrieval-augmented generation

Autor:innen und Institutionen

  • Temerty Faculty of Medicine, University of Toronto, Toronto, ON, Canada.
  • Institute of Biomedical Engineering, University of Toronto, Toronto, ON, Canada.
  • Schulich School of Medicine & Dentistry, Western University, London, ON, Canada.

ORCID-Profile: Nicholas Dietrich

Zitieren

APA

Dietrich, N., & Stubbert, B. (2025). Evaluating Adherence to Canadian Radiology Guidelines for Incidental Hepatobiliary Findings Using RAG-Enabled LLMs. Canadian Association of Radiologists Journal, 76(4), 674-682. https://doi.org/10.1177/08465371251323124

Vancouver

Dietrich N, Stubbert B. Evaluating Adherence to Canadian Radiology Guidelines for Incidental Hepatobiliary Findings Using RAG-Enabled LLMs. Canadian Association of Radiologists Journal. 2025;76(4):674-682. doi:10.1177/08465371251323124

BibTeX
@article{Dietrich2025,
  author = {Dietrich, Nicholas and Stubbert, Brett},
  title = {Evaluating Adherence to Canadian Radiology Guidelines for Incidental Hepatobiliary Findings Using RAG-Enabled LLMs},
  journal = {Canadian Association of Radiologists Journal},
  year = {2025},
  volume = {76},
  number = {4},
  pages = {674--682},
  doi = {10.1177/08465371251323124}
}
Kostenlos starten →