MulFS-CAP: Multimodal Fusion-Supervised Cross-Modality Alignment Perception for Unregistered Infrared-Visible Image Fusion
IEEE Transactions on Pattern Analysis and Machine Intelligence · 2025-05 · 47 (5) · 3673-3690
DOI: 10.1109/tpami.2025.3535617 ↗Kurz gefasst
Fachartikel in der Zeitschrift IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025-05. Verfasst von Huafeng Li und 5 weiteren Autor:innen. Die Arbeit wurde bisher 140-mal zitiert (Crossref) und nennt 57 Literaturangaben. Gefördert von: National Natural Science Foundation of China, Yunnan Fundamental Research Projects.
Abstract (Originalsprache)
In this study, we propose Multimodal Fusion-supervised Cross-modality Alignment Perception (MulFS-CAP), a novel framework for single-stage fusion of unregistered infrared-visible images. Traditional two-stage methods depend on explicit registration algorithms to align source images spatially, often adding complexity. In contrast, MulFS-CAP seamlessly blends implicit registration with fusion, simplifying the process and enhancing suitability for practical applications. MulFS-CAP utilizes a shared shallow feature encoder to merge unregistered infrared-visible images in a single stage. To address the specific requirements of feature-level alignment and fusion, we develop a consistent feature learning approach via a learnable modality dictionary. This dictionary provides complementary information for unimodal features, thereby maintaining consistency between individual and fused multimodal features. As a result, MulFS-CAP effectively reduces the impact of modality variance on cross-modality feature alignment, allowing for simultaneous registration and fusion. Additionally, in MulFS-CAP, we advance a novel cross-modality alignment approach, creating a correlation matrix to detail pixel relationships between source images. This matrix aids in aligning features across infrared and visible images, further refining the fusion process. The above designs make MulFS-CAP more lightweight, effective and explicit registration-free. Experimental results from different datasets demonstrate the effectiveness of our proposed method and its superiority over the state-of-the-art two-stage methods.
Förderung
- National Natural Science Foundation of China
- Yunnan Fundamental Research Projects
Zitieren
APA
Li, H., Yang, Z., Zhang, Y., Jia, W., Yu, Z., & Liu, Y. (2025). MulFS-CAP: Multimodal Fusion-Supervised Cross-Modality Alignment Perception for Unregistered Infrared-Visible Image Fusion. IEEE Transactions on Pattern Analysis and Machine Intelligence, 47(5), 3673-3690. https://doi.org/10.1109/tpami.2025.3535617
Vancouver
Li H, Yang Z, Zhang Y, Jia W, Yu Z, Liu Y. MulFS-CAP: Multimodal Fusion-Supervised Cross-Modality Alignment Perception for Unregistered Infrared-Visible Image Fusion. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2025;47(5):3673-3690. doi:10.1109/tpami.2025.3535617
BibTeX
@article{Li2025,
author = {Li, Huafeng and Yang, Zengyi and Zhang, Yafei and Jia, Wei and Yu, Zhengtao and Liu, Yu},
title = {MulFS-CAP: Multimodal Fusion-Supervised Cross-Modality Alignment Perception for Unregistered Infrared-Visible Image Fusion},
journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence},
year = {2025},
volume = {47},
number = {5},
pages = {3673--3690},
doi = {10.1109/tpami.2025.3535617}
}
EUVIMED