MIFNet: Learning Modality-Invariant Features for Generalizable Multimodal Image Matching
IEEE Transactions on Image Processing · 2025 · 34 · 3593-3608
DOI: 10.1109/tip.2025.3574937 ↗Kurz gefasst
Fachartikel in der Zeitschrift IEEE Transactions on Image Processing, 2025. Verfasst von Yepeng Liu und 6 weiteren Autor:innen. Die Arbeit wurde bisher 16-mal zitiert (Crossref) und nennt 70 Literaturangaben. Gefördert von: National Key Research and Development Program of China, National Natural Science Foundation of China, Innovative Research Group Project of Hubei Province.
Abstract (Originalsprache)
Many keypoint detection and description methods have been proposed for image matching or registration. While these methods demonstrate promising performance for single-modality image matching, they often struggle with multimodal data because the descriptors trained on single-modality data tend to lack robustness against the non-linear variations present in multimodal data. Extending such methods to multimodal image matching often requires well-aligned multimodal data to learn modality-invariant descriptors. However, acquiring such data is often costly and impractical in many real-world scenarios. To address this challenge, we propose a modality-invariant feature learning network (MIFNet) to compute modality-invariant features for keypoint descriptions in multimodal image matching using only single-modality training data. Specifically, we propose a novel latent feature aggregation module and a cumulative hybrid aggregation module to enhance the base keypoint descriptors trained on single-modality data by leveraging pre-trained features from Stable Diffusion models. We validate our method with recent keypoint detection and description methods in three multimodal retinal image datasets (CF-FA, CF-OCT, EMA-OCTA) and two remote sensing datasets (Optical-SAR and Optical-NIR). Extensive experiments demonstrate that the proposed MIFNet is able to learn modality-invariant feature for multimodal image matching without accessing the targeted modality and has good zero-shot generalization ability. The code will be released at https://github.com/lyp-deeplearning/MIFNet.
Förderung
- National Key Research and Development Program of China
- National Natural Science Foundation of China
- Innovative Research Group Project of Hubei Province
- Key Research and Development Program of Zhejiang Province
- Agency for Science, Technology and Research (A*STAR) through the Coastal Protection and Flood Management Research Programme
Zitieren
APA
Liu, Y., Sun, Z., Yu, B., Zhao, Y., Du, B., Xu, Y., & Cheng, J. (2025). MIFNet: Learning Modality-Invariant Features for Generalizable Multimodal Image Matching. IEEE Transactions on Image Processing, 34, 3593-3608. https://doi.org/10.1109/tip.2025.3574937
Vancouver
Liu Y, Sun Z, Yu B, Zhao Y, Du B, Xu Y, et al. MIFNet: Learning Modality-Invariant Features for Generalizable Multimodal Image Matching. IEEE Transactions on Image Processing. 2025;34:3593-3608. doi:10.1109/tip.2025.3574937
BibTeX
@article{Liu2025,
author = {Liu, Yepeng and Sun, Zhichao and Yu, Baosheng and Zhao, Yitian and Du, Bo and Xu, Yongchao and Cheng, Jun},
title = {MIFNet: Learning Modality-Invariant Features for Generalizable Multimodal Image Matching},
journal = {IEEE Transactions on Image Processing},
year = {2025},
volume = {34},
pages = {3593--3608},
doi = {10.1109/tip.2025.3574937}
}
EUVIMED