An Explainable Multimodal Framework for Breast Ultrasound Report Generation Using Vision-Language Transformers
Journal of Imaging · 27.07.2026 · 12 (8) · 338
DOI: 10.3390/jimaging12080338 ↗Kurz gefasst
Fachartikel in der Zeitschrift Journal of Imaging, online erschienen am 27.07.2026. Verfasst von Prashanth Gowda Attahalli Shivakumar und 2 weiteren Autor:innen. Beteiligte Einrichtung: Department of Computer Science and Digital Technologies, School of Architecture, Computing, and Engineering, University of East London, London E16 2R…. Die Arbeit nennt 37 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
Breast cancer remains one of the leading causes of cancer-related mortality among women worldwide, where early and accurate diagnosis is critical for effective treatment. Although recent advances in deep learning have enabled automated radiology report generation from breast ultrasound images, most existing approaches function as black-box systems, limiting clinical trust and interpretability. This study proposes a trustworthy and explainable framework for automated breast ultrasound report generation that combines Vision-Language Modelling (VLM) with multi-level Explainable Artificial Intelligence (XAI). The proposed architecture integrates a Swin Transformer for visual feature extraction, BioBERT/ClinicalBERT for clinical text representation, and a GPT-2-based decoder for report generation through a dual cross-attention fusion mechanism. The framework is evaluated on benchmark breast ultrasound datasets paired with expert-annotated radiology reports using standard natural language generation metrics, including BLEU, ROUGE-L, METEOR, and CIDEr. Experimental results demonstrate that the multimodal architecture significantly improves report quality, clinical consistency, and semantic accuracy compared with conventional image-only and single-modal baselines. To address transparency and trustworthiness, the framework provides dual-level explanations through Grad-CAM visual heatmaps and LIME/SHAP-based token attribution analysis, enabling clinicians to understand both image regions and textual features influencing generated reports. Qualitative assessment further indicates strong alignment between model explanations and radiologist-identified diagnostic findings.
Schlagwörter
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Department of Computer Science and Digital Technologies, School of Architecture, Computing, and Engineering, University of East London, London E16 2RD, UK.
ORCID-Profile: Prashanth Gowda Attahalli Shivakumar, Azhar Mahmood, Shaheen Khatoon
Zitieren
APA
Attahalli Shivakumar, P. G., Mahmood, A., & Khatoon, S. (2026). An Explainable Multimodal Framework for Breast Ultrasound Report Generation Using Vision-Language Transformers. Journal of Imaging, 12(8), 338. https://doi.org/10.3390/jimaging12080338
Vancouver
Attahalli Shivakumar PG, Mahmood A, Khatoon S. An Explainable Multimodal Framework for Breast Ultrasound Report Generation Using Vision-Language Transformers. Journal of Imaging. 2026;12(8):338. doi:10.3390/jimaging12080338
BibTeX
@article{AttahalliShivakumar2026,
author = {Attahalli Shivakumar, Prashanth Gowda and Mahmood, Azhar and Khatoon, Shaheen},
title = {An Explainable Multimodal Framework for Breast Ultrasound Report Generation Using Vision-Language Transformers},
journal = {Journal of Imaging},
year = {2026},
volume = {12},
number = {8},
pages = {338},
doi = {10.3390/jimaging12080338}
}
EUVIMED