MGFusion: a multimodal large language model-guided information perception for infrared and visible image fusion
Frontiers in Neurorobotics · 23.12.2024 · 18
DOI: 10.3389/fnbot.2024.1521603 ↗Kurz gefasst
Fachartikel in der Zeitschrift Frontiers in Neurorobotics, online erschienen am 23.12.2024. Verfasst von Zengyi Yang und 3 weiteren Autor:innen. Beteiligte Einrichtung: Faculty of Information Engineering and Automation, Kunming University of Science and Technology, Kunming, Yunnan, China. Die Arbeit wurde bisher 17-mal zitiert (Crossref) und nennt 64 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
Existing image fusion methods primarily focus on complex network structure designs while neglecting the limitations of simple fusion strategies in complex scenarios. To address this issue, this study proposes a new method for infrared and visible image fusion based on a multimodal large language model. The method proposed in this paper fully considers the high demand for semantic information in enhancing image quality as well as the fusion strategies in complex scenes. We supplement the features in the fusion network with information from the multimodal large language model and construct a new fusion strategy. To achieve this goal, we design CLIP-driven Information Injection (CII) approach and CLIP-guided Feature Fusion (CFF) strategy. CII utilizes CLIP to extract robust image features rich in semantic information, which serve to supplement the information of infrared and visible features, thereby enhancing their representation capabilities for the scene. CFF further utilizes the robust image features extracted by CLIP to select and fuse the infrared and visible features after the injection of semantic information, addressing the challenges of image fusion in complex scenes. Compared to existing methods, the main advantage of the proposed method lies in leveraging the powerful semantic understanding capabilities of the multimodal large language model to supplement information for infrared and visible features, thus avoiding the need for complex network structure designs. Experimental results on multiple public datasets validate the effectiveness and superiority of the proposed method.
Schlagwörter
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Faculty of Information Engineering and Automation, Kunming University of Science and Technology, Kunming, Yunnan, China.
- Kunming Cigarette Factory, Hongyunhonghe Tobacco Group Company Limited, Kunming, Yunnan, China.
Zitieren
APA
Yang, Z., Li, Y., Tang, X., & Xie, M. (2024). MGFusion: a multimodal large language model-guided information perception for infrared and visible image fusion. Frontiers in Neurorobotics, 18. https://doi.org/10.3389/fnbot.2024.1521603
Vancouver
Yang Z, Li Y, Tang X, Xie M. MGFusion: a multimodal large language model-guided information perception for infrared and visible image fusion. Frontiers in Neurorobotics. 2024;18. doi:10.3389/fnbot.2024.1521603
BibTeX
@article{Yang2024,
author = {Yang, Zengyi and Li, Yunping and Tang, Xin and Xie, MingHong},
title = {MGFusion: a multimodal large language model-guided information perception for infrared and visible image fusion},
journal = {Frontiers in Neurorobotics},
year = {2024},
volume = {18},
doi = {10.3389/fnbot.2024.1521603}
}
EUVIMED