EUVIMEDEUVIMED entdecken
Crossref

MulFS-CAP: Multimodal Fusion-Supervised Cross-Modality Alignment Perception for Unregistered Infrared-Visible Image Fusion

Huafeng Li, Zengyi Yang, Yafei Zhang, Wei Jia, Zhengtao Yu, Yu Liu

IEEE Transactions on Pattern Analysis and Machine Intelligence · 2025-05 · 47 (5) · 3673-3690

DOI: 10.1109/tpami.2025.3535617 ↗

Kurz gefasst

Fachartikel in der Zeitschrift IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025-05. Verfasst von Huafeng Li und 5 weiteren Autor:innen. Die Arbeit wurde bisher 140-mal zitiert (Crossref) und nennt 57 Literaturangaben. Gefördert von: National Natural Science Foundation of China, Yunnan Fundamental Research Projects.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

In this study, we propose Multimodal Fusion-supervised Cross-modality Alignment Perception (MulFS-CAP), a novel framework for single-stage fusion of unregistered infrared-visible images. Traditional two-stage methods depend on explicit registration algorithms to align source images spatially, often adding complexity. In contrast, MulFS-CAP seamlessly blends implicit registration with fusion, simplifying the process and enhancing suitability for practical applications. MulFS-CAP utilizes a shared shallow feature encoder to merge unregistered infrared-visible images in a single stage. To address the specific requirements of feature-level alignment and fusion, we develop a consistent feature learning approach via a learnable modality dictionary. This dictionary provides complementary information for unimodal features, thereby maintaining consistency between individual and fused multimodal features. As a result, MulFS-CAP effectively reduces the impact of modality variance on cross-modality feature alignment, allowing for simultaneous registration and fusion. Additionally, in MulFS-CAP, we advance a novel cross-modality alignment approach, creating a correlation matrix to detail pixel relationships between source images. This matrix aids in aligning features across infrared and visible images, further refining the fusion process. The above designs make MulFS-CAP more lightweight, effective and explicit registration-free. Experimental results from different datasets demonstrate the effectiveness of our proposed method and its superiority over the state-of-the-art two-stage methods.

Förderung

  • National Natural Science Foundation of China
  • Yunnan Fundamental Research Projects

Zitieren

APA

Li, H., Yang, Z., Zhang, Y., Jia, W., Yu, Z., & Liu, Y. (2025). MulFS-CAP: Multimodal Fusion-Supervised Cross-Modality Alignment Perception for Unregistered Infrared-Visible Image Fusion. IEEE Transactions on Pattern Analysis and Machine Intelligence, 47(5), 3673-3690. https://doi.org/10.1109/tpami.2025.3535617

Vancouver

Li H, Yang Z, Zhang Y, Jia W, Yu Z, Liu Y. MulFS-CAP: Multimodal Fusion-Supervised Cross-Modality Alignment Perception for Unregistered Infrared-Visible Image Fusion. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2025;47(5):3673-3690. doi:10.1109/tpami.2025.3535617

BibTeX
@article{Li2025,
  author = {Li, Huafeng and Yang, Zengyi and Zhang, Yafei and Jia, Wei and Yu, Zhengtao and Liu, Yu},
  title = {MulFS-CAP: Multimodal Fusion-Supervised Cross-Modality Alignment Perception for Unregistered Infrared-Visible Image Fusion},
  journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence},
  year = {2025},
  volume = {47},
  number = {5},
  pages = {3673--3690},
  doi = {10.1109/tpami.2025.3535617}
}
Kostenlos starten →