Robust multi-modal fusion architecture for medical data with knowledge distillation
Computer Methods and Programs in Biomedicine · 2025-03 · 260 · 108568
DOI: 10.1016/j.cmpb.2024.108568 ↗Kurz gefasst
Fachartikel in der Zeitschrift Computer Methods and Programs in Biomedicine, 2025-03. Verfasst von Muyu Wang und 5 weiteren Autor:innen. Beteiligte Einrichtung: School of Biomedical Engineering, Capital Medical University, No.10, Xitoutiao, You An Men, Fengtai District, Beijing 100069, China; Beijing Key Labo…. Hauptthemen laut MEDLINE: Deep Learning. Die Arbeit wurde bisher 12-mal zitiert (Crossref) und nennt 67 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: National Natural Science Foundation of China, Beijing Natural Science Foundation.
Abstract (Originalsprache)
BACKGROUND: The fusion of multi-modal data has been shown to significantly enhance the performance of deep learning models, particularly on medical data. However, missing modalities are common in medical data due to patient specificity, which poses a substantial challenge to the application of these models. OBJECTIVE: This study aimed to develop a novel and efficient multi-modal fusion framework for medical datasets that maintains consistent performance, even in the absence of one or more modalities. METHODS: In this paper, we fused three modalities: chest X-ray radiographs, history of present illness text, and tabular data such as demographics and laboratory tests. A multi-modal fusion module based on pooled bottleneck (PB) attention was proposed in conjunction with knowledge distillation (KD) for enhancing model inference in the case of missing modalities. In addition, we introduced a gradient modulation (GM) method to deal with the unbalanced optimization in multi-modal model training. Finally, we designed comparison and ablation experiments to evaluate the fusion effect, the model robustness to missing modalities, and the contribution of each component (PB, KD, and GM). The evaluation experiments were performed on the MIMIC-IV datasets with the task of predicting in-hospital mortality risk. Model performance was assessed using the area under the receiver operating characteristic curve (AUROC) and the area under the precision-recall curve (AUPRC). RESULTS: The proposed multi-modal fusion framework achieved an AUROC of 0.886 and AUPRC of 0.459, significantly surpassing the performance of baseline models. Even when one or two modalities were missing, our model consistently outperformed the reference models. Ablation of each of the three components resulted in varying degrees of performance degradation, highlighting their distinct contributions to the model's overall effectiveness. CONCLUSIONS: This innovative multi-modal fusion architecture has demonstrated robustness to missing modalities, and has shown excellent performance in fusing three medical modalities for patient outcome prediction. This study provides a novel idea for addressing the challenge of missing modalities and has the potential be scaled to additional modalities.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- School of Biomedical Engineering, Capital Medical University, No.10, Xitoutiao, You An Men, Fengtai District, Beijing 100069, China; Beijing Key Laboratory of Fundamental Research on Biomechanics in Clinical Application…
Förderung
- National Natural Science Foundation of China
- Beijing Natural Science Foundation
Zitieren
APA
Wang, M., Fan, S., Li, Y., Gao, B., Xie, Z., & Chen, H. (2025). Robust multi-modal fusion architecture for medical data with knowledge distillation. Computer Methods and Programs in Biomedicine, 260, 108568. https://doi.org/10.1016/j.cmpb.2024.108568
Vancouver
Wang M, Fan S, Li Y, Gao B, Xie Z, Chen H. Robust multi-modal fusion architecture for medical data with knowledge distillation. Computer Methods and Programs in Biomedicine. 2025;260:108568. doi:10.1016/j.cmpb.2024.108568
BibTeX
@article{Wang2025,
author = {Wang, Muyu and Fan, Shiyu and Li, Yichen and Gao, Binyu and Xie, Zhongrang and Chen, Hui},
title = {Robust multi-modal fusion architecture for medical data with knowledge distillation},
journal = {Computer Methods and Programs in Biomedicine},
year = {2025},
volume = {260},
pages = {108568},
doi = {10.1016/j.cmpb.2024.108568}
}
EUVIMED