EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

An Explainable Multimodal Framework for Breast Ultrasound Report Generation Using Vision-Language Transformers

Prashanth Gowda Attahalli Shivakumar iD, Azhar Mahmood iD, Shaheen Khatoon iD

Journal of Imaging · 27.07.2026 · 12 (8) · 338

DOI: 10.3390/jimaging12080338 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Journal of Imaging, online erschienen am 27.07.2026. Verfasst von Prashanth Gowda Attahalli Shivakumar und 2 weiteren Autor:innen. Beteiligte Einrichtung: Department of Computer Science and Digital Technologies, School of Architecture, Computing, and Engineering, University of East London, London E16 2R…. Die Arbeit nennt 37 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Breast cancer remains one of the leading causes of cancer-related mortality among women worldwide, where early and accurate diagnosis is critical for effective treatment. Although recent advances in deep learning have enabled automated radiology report generation from breast ultrasound images, most existing approaches function as black-box systems, limiting clinical trust and interpretability. This study proposes a trustworthy and explainable framework for automated breast ultrasound report generation that combines Vision-Language Modelling (VLM) with multi-level Explainable Artificial Intelligence (XAI). The proposed architecture integrates a Swin Transformer for visual feature extraction, BioBERT/ClinicalBERT for clinical text representation, and a GPT-2-based decoder for report generation through a dual cross-attention fusion mechanism. The framework is evaluated on benchmark breast ultrasound datasets paired with expert-annotated radiology reports using standard natural language generation metrics, including BLEU, ROUGE-L, METEOR, and CIDEr. Experimental results demonstrate that the multimodal architecture significantly improves report quality, clinical consistency, and semantic accuracy compared with conventional image-only and single-modal baselines. To address transparency and trustworthiness, the framework provides dual-level explanations through Grad-CAM visual heatmaps and LIME/SHAP-based token attribution analysis, enabling clinicians to understand both image regions and textual features influencing generated reports. Qualitative assessment further indicates strong alignment between model explanations and radiologist-identified diagnostic findings.

Schlagwörter

Stichwörter der Autor:innen

ClinicalBERTGPT-2automatic report generationbreast ultrasoundmultimodal learningswin transformer

Autor:innen und Institutionen

  • Department of Computer Science and Digital Technologies, School of Architecture, Computing, and Engineering, University of East London, London E16 2RD, UK.

ORCID-Profile: Prashanth Gowda Attahalli Shivakumar, Azhar Mahmood, Shaheen Khatoon

Zitieren

APA

Attahalli Shivakumar, P. G., Mahmood, A., & Khatoon, S. (2026). An Explainable Multimodal Framework for Breast Ultrasound Report Generation Using Vision-Language Transformers. Journal of Imaging, 12(8), 338. https://doi.org/10.3390/jimaging12080338

Vancouver

Attahalli Shivakumar PG, Mahmood A, Khatoon S. An Explainable Multimodal Framework for Breast Ultrasound Report Generation Using Vision-Language Transformers. Journal of Imaging. 2026;12(8):338. doi:10.3390/jimaging12080338

BibTeX
@article{AttahalliShivakumar2026,
  author = {Attahalli Shivakumar, Prashanth Gowda and Mahmood, Azhar and Khatoon, Shaheen},
  title = {An Explainable Multimodal Framework for Breast Ultrasound Report Generation Using Vision-Language Transformers},
  journal = {Journal of Imaging},
  year = {2026},
  volume = {12},
  number = {8},
  pages = {338},
  doi = {10.3390/jimaging12080338}
}
Kostenlos starten →