Multi-Modal Vision–Language Transformer for Integrating Nano-Resolution Medical Imaging and Clinical Text in Virtual Diagnostic Assistants
Journal of Nano Molecular Intelligence and Virtual Health Systems · 2025-06-30 · 1 (2)
DOI: 10.70023/nmivhs.122 ↗At a glance
Journal article in Journal of Nano Molecular Intelligence and Virtual Health Systems, published online on 2025-06-30. Written by Mariam Fawzy Kamel and 1 co-authors.
Abstract (original language)
Nano-resolution medical imaging and clinical text integration still face significant challenges in intelligent healthcare due to the heterogeneity of data representations, the lack of cross-modal interaction, and inadequate interpretability in diagnosis. Most diagnostic systems available today manage visual and textual data separately, resulting in limited clinical understanding and reduced diagnostic certainty. In this paper, a new Hierarchical Cross-Attention Vision–Language Transformer with Graph Knowledge Fusion (HCVLT-GKF) is proposed to address these drawbacks in multimodal disease diagnosis in virtual diagnostic assistants. The proposed framework integrates two approaches: a Vision–Language Transformer (VLT) that uses hierarchical cross-attention to learn unified representations from nano-resolution medical images and clinical narratives, and a Graph Knowledge Fusion (GKF) module that incorporates structured medical knowledge to model relationships among symptoms, biomarkers, and diseases. The VLT contributes to the semantic congruence between the visual and textual modalities, whereas the GKF module can help improve the ability to make diagnostic deductions and enable contextual understanding. Experimental validation shows that the proposed HCVLT-GKF achieves 99.12% accuracy, 98.94% precision, 98.85% recall, 98.89% F1-score, and 99.28% AUC, which are better than those of existing multimodal diagnostic models. Moreover, the framework offers better interpretability through knowledge-guided reasoning. The findings show that HCVLT-GKF provides an effective and understandable approach to multimodal clinical decision-making, supporting and enabling the design of next-generation virtual diagnostic assistants for precision healthcare.
Cite
APA
Kamel, M. F., & Rahimi, N. (2025). Multi-Modal Vision–Language Transformer for Integrating Nano-Resolution Medical Imaging and Clinical Text in Virtual Diagnostic Assistants. Journal of Nano Molecular Intelligence and Virtual Health Systems, 1(2). https://doi.org/10.70023/nmivhs.122
Vancouver
Kamel MF, Rahimi N. Multi-Modal Vision–Language Transformer for Integrating Nano-Resolution Medical Imaging and Clinical Text in Virtual Diagnostic Assistants. Journal of Nano Molecular Intelligence and Virtual Health Systems. 2025;1(2). doi:10.70023/nmivhs.122
BibTeX
@article{Kamel2025,
author = {Kamel, Mariam Fawzy and Rahimi, Neda},
title = {Multi-Modal Vision–Language Transformer for Integrating Nano-Resolution Medical Imaging and Clinical Text in Virtual Diagnostic Assistants},
journal = {Journal of Nano Molecular Intelligence and Virtual Health Systems},
year = {2025},
volume = {1},
number = {2},
doi = {10.70023/nmivhs.122}
}
EUVIMED