ViT-MultiRAGNet: A scalable and reliable retrieval-augmented Vision Transformer framework for memory-guided feature fusion multi-modal mammogram classification
PLOS One · 15.07.2026 · 21 (7) · e0349865
DOI: 10.1371/journal.pone.0349865 ↗Kurz gefasst
Fachartikel in der Zeitschrift PLOS One, online erschienen am 15.07.2026. Verfasst von N Thirupathi Rao und 4 weiteren Autor:innen. Beteiligte Einrichtung: Department of Computer Science and Engineering, Vignan's Institute of Information Technology (A), Visakhapatnam, Andhra Pradesh, India. Hauptthemen laut MEDLINE: Mammography, Breast Neoplasms. Die Arbeit nennt 19 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: SERB, DST, Government of India, Princess Nourah Bint Abdulrahman University.
Abstract (Originalsprache)
Background
Diagnosing breast cancer with mammography continues to be an incidentally challenging process because of the numerous different ways of acquiring the images, the vast difference in appearance between lesions, and the minimal use of historical information regarding patients being considered when making a diagnosis. In addition, many current deep learning algorithms use static weight values based on the most recent image taken only, often failing to relate a current case to similar historical examples and to incorporate retrieval-based contextual evidence during inference.
Introduction
ViT-MultiRAGNet is a retrieval-augmented Vision Transformer framework designed to integrate multi-view 2D mammographic features with memory-bank-based contextual evidence from historically similar cases. The proposed approach enhances diagnostic accuracy by combining global 2D mammographic features, semantic lesion representations, and historical case evidence retrieved from a fold-wise memory bank during inference.
Methods
Multi-view 2D mammograms were processed using a Vision Transformer encoder to extract global contextual features, while a retrieval-augmented memory bank was used to identify diagnostically similar historical cases for evidence-guided feature enrichment. A Retrieval-Augmented Generation (RAG) module retrieves the top-k most similar feature embeddings from a fold-wise training memory bank and combines them with the current query representation using multi-head cross-attention. The evaluation of the models included utilizing accuracy, area under the receiver operating curve (AUC-ROC), Dice Coefficient, and F1 score metrics at a 95% bootstrapped confidence interval using five-fold stratified cross-validation, with findings assessed using Wilcoxon signed-rank statistical tests.
Results
The accuracy of ViT-MultiRAGNet is calculated as a value of 0.978 ± 0.009 and AUC-ROC 0.998 ± 0.009 (Wilcoxon p < 0.01), which is significantly higher than unimodal benchmarks and fusion techniques on the RTM dataset. The calculations for the CBIS-DDSM evaluation were 0.961 ± 0.011 for accuracy and 0.989 ± 0.010 for AUC-ROC. The calculations for segmentation evaluation were 0.882 (RTM) and 0.795 (CBIS-DDSM), however both have a higher degree of alignment at the edge. The RAG mechanism improved retrieval-guided fusion compared with simple feature concatenation while maintaining efficient inference performance of 0.31 seconds per image.
Conclusion
The combination of ViT-based global 2D mammographic representation, retrieval-augmented fusion, and memory-guided contextual evidence improves diagnostic performance, robustness, and clinical interpretability. This Framework appears to be a good candidate for an Evidence-based Decision Support System which will provide a transparent method of helping clinicians make decisions regarding screening and diagnosis of breast cancer.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Autor:innen und Institutionen
- Department of Computer Science and Engineering, Vignan's Institute of Information Technology (A), Visakhapatnam, Andhra Pradesh, India.
- Vignan's Institute of Information Technology (A), Visakhapatnam, Andhra Pradesh, India.
- Department of Computer Sciences, College of Computer and Information Sciences, Princess Nourah bint Abdulrahman University, Riyadh, Saudi Arabia.
- Department of Software Engineering, College of Computer Engineering and Science, Prince Mohammad Bin Fahd University, Al Khobar, Kingdom of Saudi Arabia.
- Computer Science Department, Faculty of Computers and Information, Qena University, Qena, Egypt.
ORCID-Profile: N Thirupathi Rao, Ch V V Ramana, Ayman Aljarbouh, Samih M Mostafa
Förderung
- SERB, DST, Government of India
- Princess Nourah Bint Abdulrahman University
Zitieren
APA
Rao, N. T., Ramana, C. V. V., Karim, F. K., Aljarbouh, A., & Mostafa, S. M. (2026). ViT-MultiRAGNet: A scalable and reliable retrieval-augmented Vision Transformer framework for memory-guided feature fusion multi-modal mammogram classification. PLOS One, 21(7), e0349865. https://doi.org/10.1371/journal.pone.0349865
Vancouver
Rao NT, Ramana CVV, Karim FK, Aljarbouh A, Mostafa SM. ViT-MultiRAGNet: A scalable and reliable retrieval-augmented Vision Transformer framework for memory-guided feature fusion multi-modal mammogram classification. PLOS One. 2026;21(7):e0349865. doi:10.1371/journal.pone.0349865
BibTeX
@article{Rao2026,
author = {Rao, N Thirupathi and Ramana, Ch V V and Karim, Faten Khalid and Aljarbouh, Ayman and Mostafa, Samih M},
title = {ViT-MultiRAGNet: A scalable and reliable retrieval-augmented Vision Transformer framework for memory-guided feature fusion multi-modal mammogram classification},
journal = {PLOS One},
year = {2026},
volume = {21},
number = {7},
pages = {e0349865},
doi = {10.1371/journal.pone.0349865}
}
EUVIMED