EUVIMEDEUVIMED entdecken
Open Access

Can Small Open-Source Language Models With Retrieval-Augmented Generation Match GPT-4 Performance in Breast Cancer Clinical Decision Support?

Chanhee Park iD, In Hae Park iD, Minhyuk Kim, Youngjoon Jang iD, Heuiseok Lim iD, Yuna Hur iD

JCO Clinical Cancer Informatics · 2026-04 · 10 (2)

DOI: 10.1200/cci-25-00361 ↗

Kurz gefasst

Fachartikel in der Zeitschrift JCO Clinical Cancer Informatics, 2026-04. Verfasst von Chanhee Park und 5 weiteren Autor:innen. Beteiligte Einrichtung: Department of Computer Science and Engineering, Korea University, Seoul, Korea. Hauptthemen laut MEDLINE: Breast Neoplasms, Decision Support Systems, Clinical. Die Arbeit nennt 50 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

PURPOSE The rapidly evolving breast cancer treatment landscape creates significant information synthesis challenges for clinicians. We evaluated whether small open-source large language models (LLMs) augmented with retrieval-augmented generation (RAG) could match proprietary model performance for clinical guideline queries. METHODS We developed a domain-specialized RAG pipeline using HTML-structure-preserving chunking of 1,356 ASCO breast cancer guideline documents. Five LLMs were each evaluated with and without RAG: GPT-4-turbo, GPT-3.5-turbo, Qwen2.5-14B (14 billion parameters), LLaMA3-8B, and OpenBioLLM-8B. Performance was assessed using 98 expert-curated question-answer-context triplets across seven breast cancer categories. Evaluation used both rubric-based scoring (six metrics: fluency, relevance, reliability, consistency, clarity, and clinical impact) and exhaustive pairwise ranking by GPT-4-turbo as judge. Human validation was conducted with 15 practicing oncologists on a 10-query subset. RESULTS RAG-enhanced Qwen2.5-14B achieved mean rubric scores of 3.77 versus 3.96 for GPT-4-turbo and pairwise ranking performance of 0.72 versus 0.81 (normalized scale). Although absolute rubric gains were modest (0.02-0.05 on a five-point scale), relative improvements in head-to-head win rates ranged from 16% to 46%. Human expert scores confirmed RAG superiority but were consistently more conservative than LLM judge scores (mean 3.81 v 4.12 across all metrics). Optimal retrieval used top-5 contexts; performance degraded sharply at higher context volumes. CONCLUSION Small open-source LLMs with optimized RAG can approach state-of-the-art proprietary model performance for clinical decision support. This approach enables scalable, cost-effective, privacy-preserving deployment without recurrent fine-tuning, suggesting potential for real-world clinical implementation on single-graphics processing unit infrastructure under expert supervision.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Breast NeoplasmsDecision Support Systems, ClinicalHumansFemaleLarge Language ModelsGenerative Artificial IntelligenceReproducibility of Results

Autor:innen und Institutionen

  • Department of Computer Science and Engineering, Korea University, Seoul, Korea.
  • Department of Hemato-Oncology, Korea University College of Medicine, Seoul, Korea.
  • Human-Inspired AI Research, Seoul, Korea.

ORCID-Profile: Chanhee Park, In Hae Park, Youngjoon Jang, Heuiseok Lim, Yuna Hur

Zitieren

APA

Park, C., Park, I. H., Kim, M., Jang, Y., Lim, H., & Hur, Y. (2026). Can Small Open-Source Language Models With Retrieval-Augmented Generation Match GPT-4 Performance in Breast Cancer Clinical Decision Support?. JCO Clinical Cancer Informatics, 10(2). https://doi.org/10.1200/cci-25-00361

Vancouver

Park C, Park IH, Kim M, Jang Y, Lim H, Hur Y. Can Small Open-Source Language Models With Retrieval-Augmented Generation Match GPT-4 Performance in Breast Cancer Clinical Decision Support?. JCO Clinical Cancer Informatics. 2026;10(2). doi:10.1200/cci-25-00361

BibTeX
@article{Park2026,
  author = {Park, Chanhee and Park, In Hae and Kim, Minhyuk and Jang, Youngjoon and Lim, Heuiseok and Hur, Yuna},
  title = {Can Small Open-Source Language Models With Retrieval-Augmented Generation Match GPT-4 Performance in Breast Cancer Clinical Decision Support?},
  journal = {JCO Clinical Cancer Informatics},
  year = {2026},
  volume = {10},
  number = {2},
  doi = {10.1200/cci-25-00361}
}
Kostenlos starten →