EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

ViT-MultiRAGNet: A scalable and reliable retrieval-augmented Vision Transformer framework for memory-guided feature fusion multi-modal mammogram classification

N Thirupathi Rao iD, Ch V V Ramana iD, Faten Khalid Karim, Ayman Aljarbouh iD, Samih M Mostafa iD

PLOS One · 15.07.2026 · 21 (7) · e0349865

DOI: 10.1371/journal.pone.0349865 ↗

Kurz gefasst

Fachartikel in der Zeitschrift PLOS One, online erschienen am 15.07.2026. Verfasst von N Thirupathi Rao und 4 weiteren Autor:innen. Beteiligte Einrichtung: Department of Computer Science and Engineering, Vignan's Institute of Information Technology (A), Visakhapatnam, Andhra Pradesh, India. Hauptthemen laut MEDLINE: Mammography, Breast Neoplasms. Die Arbeit nennt 19 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: SERB, DST, Government of India, Princess Nourah Bint Abdulrahman University.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

Diagnosing breast cancer with mammography continues to be an incidentally challenging process because of the numerous different ways of acquiring the images, the vast difference in appearance between lesions, and the minimal use of historical information regarding patients being considered when making a diagnosis. In addition, many current deep learning algorithms use static weight values based on the most recent image taken only, often failing to relate a current case to similar historical examples and to incorporate retrieval-based contextual evidence during inference.

Introduction

ViT-MultiRAGNet is a retrieval-augmented Vision Transformer framework designed to integrate multi-view 2D mammographic features with memory-bank-based contextual evidence from historically similar cases. The proposed approach enhances diagnostic accuracy by combining global 2D mammographic features, semantic lesion representations, and historical case evidence retrieved from a fold-wise memory bank during inference.

Methods

Multi-view 2D mammograms were processed using a Vision Transformer encoder to extract global contextual features, while a retrieval-augmented memory bank was used to identify diagnostically similar historical cases for evidence-guided feature enrichment. A Retrieval-Augmented Generation (RAG) module retrieves the top-k most similar feature embeddings from a fold-wise training memory bank and combines them with the current query representation using multi-head cross-attention. The evaluation of the models included utilizing accuracy, area under the receiver operating curve (AUC-ROC), Dice Coefficient, and F1 score metrics at a 95% bootstrapped confidence interval using five-fold stratified cross-validation, with findings assessed using Wilcoxon signed-rank statistical tests.

Results

The accuracy of ViT-MultiRAGNet is calculated as a value of 0.978 ± 0.009 and AUC-ROC 0.998 ± 0.009 (Wilcoxon p < 0.01), which is significantly higher than unimodal benchmarks and fusion techniques on the RTM dataset. The calculations for the CBIS-DDSM evaluation were 0.961 ± 0.011 for accuracy and 0.989 ± 0.010 for AUC-ROC. The calculations for segmentation evaluation were 0.882 (RTM) and 0.795 (CBIS-DDSM), however both have a higher degree of alignment at the edge. The RAG mechanism improved retrieval-guided fusion compared with simple feature concatenation while maintaining efficient inference performance of 0.31 seconds per image.

Conclusion

The combination of ViT-based global 2D mammographic representation, retrieval-augmented fusion, and memory-guided contextual evidence improves diagnostic performance, robustness, and clinical interpretability. This Framework appears to be a good candidate for an Evidence-based Decision Support System which will provide a transparent method of helping clinicians make decisions regarding screening and diagnosis of breast cancer.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

MammographyBreast NeoplasmsHumansFemaleAlgorithmsDeep Learning

Autor:innen und Institutionen

  • Department of Computer Science and Engineering, Vignan's Institute of Information Technology (A), Visakhapatnam, Andhra Pradesh, India.
  • Vignan's Institute of Information Technology (A), Visakhapatnam, Andhra Pradesh, India.
  • Department of Computer Sciences, College of Computer and Information Sciences, Princess Nourah bint Abdulrahman University, Riyadh, Saudi Arabia.
  • Department of Software Engineering, College of Computer Engineering and Science, Prince Mohammad Bin Fahd University, Al Khobar, Kingdom of Saudi Arabia.
  • Computer Science Department, Faculty of Computers and Information, Qena University, Qena, Egypt.

ORCID-Profile: N Thirupathi Rao, Ch V V Ramana, Ayman Aljarbouh, Samih M Mostafa

Förderung

  • SERB, DST, Government of India
  • Princess Nourah Bint Abdulrahman University

Zitieren

APA

Rao, N. T., Ramana, C. V. V., Karim, F. K., Aljarbouh, A., & Mostafa, S. M. (2026). ViT-MultiRAGNet: A scalable and reliable retrieval-augmented Vision Transformer framework for memory-guided feature fusion multi-modal mammogram classification. PLOS One, 21(7), e0349865. https://doi.org/10.1371/journal.pone.0349865

Vancouver

Rao NT, Ramana CVV, Karim FK, Aljarbouh A, Mostafa SM. ViT-MultiRAGNet: A scalable and reliable retrieval-augmented Vision Transformer framework for memory-guided feature fusion multi-modal mammogram classification. PLOS One. 2026;21(7):e0349865. doi:10.1371/journal.pone.0349865

BibTeX
@article{Rao2026,
  author = {Rao, N Thirupathi and Ramana, Ch V V and Karim, Faten Khalid and Aljarbouh, Ayman and Mostafa, Samih M},
  title = {ViT-MultiRAGNet: A scalable and reliable retrieval-augmented Vision Transformer framework for memory-guided feature fusion multi-modal mammogram classification},
  journal = {PLOS One},
  year = {2026},
  volume = {21},
  number = {7},
  pages = {e0349865},
  doi = {10.1371/journal.pone.0349865}
}
Kostenlos starten →